[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:10:43 dist-test-slave-jkbq krb5kdc[11014](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:10:43 dist-test-slave-jkbq krb5kdc[11014](info): set up 2 sockets
Jun 20 23:10:43 dist-test-slave-jkbq krb5kdc[11014](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 23:10:44.020579 11007 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Jun 20 23:10:44 dist-test-slave-jkbq krb5kdc[11014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:10:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:10:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.191.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
I20260620 23:10:44.191716 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:46343
--webserver_interface=127.10.191.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:35837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:46343
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:10:44.505513 11027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:10:44.505837 11027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:10:44.505998 11027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:10:44.513270 11027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:10:44.513381 11027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:10:44.513489 11027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:10:44.513586 11027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:10:44.513670 11027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:10:44.523439 11027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:35837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:46343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:46343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:10:44.524828 11027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:10:44.526196 11027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:10:44.538518 11032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:44.538592 11035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:44.538566 11033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:10:44.539299 11027 server_base.cc:1034] running on GCE node
I20260620 23:10:44.540123 11027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:10:44.541540 11027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:10:44.542798 11027 hybrid_clock.cc:648] HybridClock initialized: now 1781997044542741 us; error 44 us; skew 500 ppm
I20260620 23:10:44.543171 11027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:10:44 dist-test-slave-jkbq krb5kdc[11014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997044, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:10:44.551216 11027 init.cc:377] Logged in from keytab as kudu/127.10.191.254@KRBTEST.COM (short username kudu)
I20260620 23:10:44.552752 11027 webserver.cc:460] Webserver started at http://127.10.191.254:34769/ using document root <none> and password file <none>
I20260620 23:10:44.553462 11027 fs_manager.cc:359] Metadata directory not provided
I20260620 23:10:44.553607 11027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:10:44.553915 11027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:10:44.556958 11027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "3dc3660004d84e26a4e2aa5fa8e1c31f"
format_stamp: "Formatted at 2026-06-20 23:10:44 on dist-test-slave-jkbq"
server_key: "9f67b5bdd8b3c6196124e5e01016066d"
server_key_iv: "f5a341b95df66d6f095d78dee6256f8b"
server_key_version: "encryptionkey@0"
I20260620 23:10:44.557724 11027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "3dc3660004d84e26a4e2aa5fa8e1c31f"
format_stamp: "Formatted at 2026-06-20 23:10:44 on dist-test-slave-jkbq"
server_key: "9f67b5bdd8b3c6196124e5e01016066d"
server_key_iv: "f5a341b95df66d6f095d78dee6256f8b"
server_key_version: "encryptionkey@0"
I20260620 23:10:44.563090 11027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 23:10:44.566480 11042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:44.567866 11027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:10:44.568058 11027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "3dc3660004d84e26a4e2aa5fa8e1c31f"
format_stamp: "Formatted at 2026-06-20 23:10:44 on dist-test-slave-jkbq"
server_key: "9f67b5bdd8b3c6196124e5e01016066d"
server_key_iv: "f5a341b95df66d6f095d78dee6256f8b"
server_key_version: "encryptionkey@0"
I20260620 23:10:44.568285 11027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:10:44.581266 11027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:10:44.585271 11027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:10:44.585597 11027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:10:44.603902 11027 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:46343
I20260620 23:10:44.603910 11093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:46343 every 8 connection(s)
I20260620 23:10:44.605317 11027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:10:44.606784 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 11027
I20260620 23:10:44.607123 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:10:44.608955 11007 external_mini_cluster.cc:1428] Setting key b54d9f97f299ec334b0ecfca3a3c2c47
I20260620 23:10:44.609297 11094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:44.620975 11094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dc3660004d84e26a4e2aa5fa8e1c31f: Bootstrap starting.
I20260620 23:10:44.624186 11094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3dc3660004d84e26a4e2aa5fa8e1c31f: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:44.625242 11094 log.cc:824] T 00000000000000000000000000000000 P 3dc3660004d84e26a4e2aa5fa8e1c31f: Log is configured to *not* fsync() on all Append() calls
I20260620 23:10:44.627993 11094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dc3660004d84e26a4e2aa5fa8e1c31f: No bootstrap required, opened a new log
I20260620 23:10:44.635246 11094 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3dc3660004d84e26a4e2aa5fa8e1c31f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc3660004d84e26a4e2aa5fa8e1c31f" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 46343 } }
I20260620 23:10:44.635622 11094 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3dc3660004d84e26a4e2aa5fa8e1c31f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:10:44.635788 11094 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3dc3660004d84e26a4e2aa5fa8e1c31f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dc3660004d84e26a4e2aa5fa8e1c31f, State: Initialized, Role: FOLLOWER
Jun 20 23:10:44 dist-test-slave-jkbq krb5kdc[11014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:10:44.636443 11094 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3dc3660004d84e26a4e2aa5fa8e1c31f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc3660004d84e26a4e2aa5fa8e1c31f" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 46343 } }
I20260620 23:10:44.636687 11094 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3dc3660004d84e26a4e2aa5fa8e1c31f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:10:44.636830 11094 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3dc3660004d84e26a4e2aa5fa8e1c31f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:10:44.637012 11094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3dc3660004d84e26a4e2aa5fa8e1c31f [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:10:44.639765 11094 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3dc3660004d84e26a4e2aa5fa8e1c31f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc3660004d84e26a4e2aa5fa8e1c31f" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 46343 } }
I20260620 23:10:44.640293 11094 leader_election.cc:302] T 00000000000000000000000000000000 P 3dc3660004d84e26a4e2aa5fa8e1c31f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3dc3660004d84e26a4e2aa5fa8e1c31f; no voters:
I20260620 23:10:44.640967 11094 leader_election.cc:288] T 00000000000000000000000000000000 P 3dc3660004d84e26a4e2aa5fa8e1c31f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:10:44.641091 11099 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3dc3660004d84e26a4e2aa5fa8e1c31f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:10:44.642518 11099 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3dc3660004d84e26a4e2aa5fa8e1c31f [term 1 LEADER]: Becoming Leader. State: Replica: 3dc3660004d84e26a4e2aa5fa8e1c31f, State: Running, Role: LEADER
I20260620 23:10:44.643360 11094 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3dc3660004d84e26a4e2aa5fa8e1c31f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:10:44.643254 11099 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3dc3660004d84e26a4e2aa5fa8e1c31f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc3660004d84e26a4e2aa5fa8e1c31f" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 46343 } }
I20260620 23:10:44.647576 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:44.612923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37780 (local address 127.10.191.254:46343)
0620 23:10:44.613697 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:44.613708 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:44.615208 (+ 1500us) server_negotiation.cc:407] Connection header received
0620 23:10:44.623913 (+ 8705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:44.623934 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:44.624385 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:44.624771 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:44.627742 (+ 2971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:44.629171 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:44.630806 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:44.631563 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:44.631779 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:44.637590 (+ 5811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:44.637598 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:44.637613 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:44.637647 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:44.640836 (+ 3189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:44.641931 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:44.641939 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:44.641947 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:44.642156 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:44.642776 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:44.642781 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:44.642784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:44.643632 (+ 848us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:44.643912 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:44.645173 (+ 1261us) server_negotiation.cc:299] Negotiation successful
0620 23:10:44.645759 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":339,"threads_started":1}
I20260620 23:10:44.648309 11100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dc3660004d84e26a4e2aa5fa8e1c31f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3dc3660004d84e26a4e2aa5fa8e1c31f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc3660004d84e26a4e2aa5fa8e1c31f" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 46343 } } }
I20260620 23:10:44.648681 11100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dc3660004d84e26a4e2aa5fa8e1c31f [sys.catalog]: This master's current role is: LEADER
I20260620 23:10:44.649171 11101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dc3660004d84e26a4e2aa5fa8e1c31f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3dc3660004d84e26a4e2aa5fa8e1c31f. Latest consensus state: current_term: 1 leader_uuid: "3dc3660004d84e26a4e2aa5fa8e1c31f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc3660004d84e26a4e2aa5fa8e1c31f" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 46343 } } }
I20260620 23:10:44.649441 11101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dc3660004d84e26a4e2aa5fa8e1c31f [sys.catalog]: This master's current role is: LEADER
I20260620 23:10:44.660722 11106 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:10:44.666566 11106 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:10:44.675230 11106 catalog_manager.cc:1347] Generated new cluster ID: 278b491b1dcf44ba92869e7529742de5
I20260620 23:10:44.675355 11106 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:10:44.691839 11106 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:10:44.693550 11106 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:10:44.703511 11106 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3dc3660004d84e26a4e2aa5fa8e1c31f: Generated new TSK 0
I20260620 23:10:44.704397 11106 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.191.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab.
Entry for principal kudu/127.10.191.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab.
I20260620 23:10:44.760977 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.193:0
--local_ip_for_outbound_sockets=127.10.191.193
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:46343
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:35837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:10:45.087996 11121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:10:45.088270 11121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:10:45.088428 11121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:10:45.095650 11121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:10:45.095747 11121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:10:45.095965 11121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.193
I20260620 23:10:45.105338 11121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:35837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:46343
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.193
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:10:45.106561 11121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:10:45.107941 11121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:10:45.120975 11126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:45.120949 11129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:45.121013 11127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:10:45.121783 11121 server_base.cc:1034] running on GCE node
I20260620 23:10:45.122413 11121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:10:45.123422 11121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:10:45.124641 11121 hybrid_clock.cc:648] HybridClock initialized: now 1781997045124610 us; error 57 us; skew 500 ppm
I20260620 23:10:45.124946 11121 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:10:45 dist-test-slave-jkbq krb5kdc[11014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997045, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:10:45.127974 11121 init.cc:377] Logged in from keytab as kudu/127.10.191.193@KRBTEST.COM (short username kudu)
I20260620 23:10:45.129444 11121 webserver.cc:460] Webserver started at http://127.10.191.193:37967/ using document root <none> and password file <none>
I20260620 23:10:45.130100 11121 fs_manager.cc:359] Metadata directory not provided
I20260620 23:10:45.130240 11121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:10:45.130548 11121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:10:45.133507 11121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-0/data/instance:
uuid: "a310c6efde2f4e0a95ad24790416dd51"
format_stamp: "Formatted at 2026-06-20 23:10:45 on dist-test-slave-jkbq"
server_key: "2298c3ee769272a80afb6f877c4fb68c"
server_key_iv: "8649f3e46c767227640a91e5a8447b8a"
server_key_version: "encryptionkey@0"
I20260620 23:10:45.134253 11121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance:
uuid: "a310c6efde2f4e0a95ad24790416dd51"
format_stamp: "Formatted at 2026-06-20 23:10:45 on dist-test-slave-jkbq"
server_key: "2298c3ee769272a80afb6f877c4fb68c"
server_key_iv: "8649f3e46c767227640a91e5a8447b8a"
server_key_version: "encryptionkey@0"
I20260620 23:10:45.139413 11121 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 23:10:45.142604 11136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:45.144019 11121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:10:45.144196 11121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-0/wal
uuid: "a310c6efde2f4e0a95ad24790416dd51"
format_stamp: "Formatted at 2026-06-20 23:10:45 on dist-test-slave-jkbq"
server_key: "2298c3ee769272a80afb6f877c4fb68c"
server_key_iv: "8649f3e46c767227640a91e5a8447b8a"
server_key_version: "encryptionkey@0"
I20260620 23:10:45.144387 11121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:10:45.176247 11121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:10:45.180246 11121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:10:45.180528 11121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:10:45.181458 11121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:10:45.182778 11121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:10:45.182906 11121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 23:10:45.183058 11121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:10:45.183130 11121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:45.216308 11121 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.193:33107
I20260620 23:10:45.216373 11248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.193:33107 every 8 connection(s)
I20260620 23:10:45.217623 11121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
I20260620 23:10:45.218626 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 11121
I20260620 23:10:45.218883 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance
I20260620 23:10:45.219620 11007 external_mini_cluster.cc:1428] Setting key 08b2e9c45cb8588220d145ad56659ca6
Jun 20 23:10:45 dist-test-slave-jkbq krb5kdc[11014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997045, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.191.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.194@KRBTEST.COM" created.
I20260620 23:10:45.240559 11251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:45.220852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:38719 (local address 127.10.191.254:46343)
0620 23:10:45.221314 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:45.221320 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:45.222311 (+ 991us) server_negotiation.cc:407] Connection header received
0620 23:10:45.225263 (+ 2952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:45.225268 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:45.225340 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:45.225442 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:45.227322 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:45.228552 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:45.230081 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:45.230789 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:45.230995 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:45.234753 (+ 3758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:45.234781 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:45.234784 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:45.234815 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:45.237555 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:45.238280 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:45.238284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:45.238287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:45.238436 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:45.238924 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:45.238929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:45.238932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:45.239177 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:45.239327 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:45.239980 (+ 653us) server_negotiation.cc:299] Negotiation successful
0620 23:10:45.240190 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":215,"threads_started":1}
I20260620 23:10:45.241458 11252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:45.221293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:46343 (local address 127.10.191.193:38719)
0620 23:10:45.222117 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:45.222181 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:45.224963 (+ 2782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:45.225713 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:45.225726 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:45.226517 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:10:45.227148 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:45.227167 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:45.228722 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:45.228726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:45.229934 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:45.229946 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:45.231100 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:45.231105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:45.231308 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:45.232025 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:10:45.232048 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:10:45.234510 (+ 2462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:10:45.237674 (+ 3164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:45.237680 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:45.237691 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:45.238177 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:45.238565 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:45.238569 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:45.238574 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:45.238765 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:45.239301 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:10:45.239308 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:10:45.239625 (+ 317us) client_negotiation.cc:769] Sending connection context
0620 23:10:45.239891 (+ 266us) client_negotiation.cc:240] Negotiation successful
0620 23:10:45.240387 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":316,"threads_started":1}
I20260620 23:10:45.242957 11249 heartbeater.cc:347] Connected to a master server at 127.10.191.254:46343
I20260620 23:10:45.243301 11249 heartbeater.cc:464] Registering TS with master...
I20260620 23:10:45.244144 11249 heartbeater.cc:511] Master 127.10.191.254:46343 requested a full tablet report, sending...
I20260620 23:10:45.246359 11059 ts_manager.cc:194] Registered new tserver with Master: a310c6efde2f4e0a95ad24790416dd51 (127.10.191.193:33107)
I20260620 23:10:45.248050 11059 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.193@KRBTEST.COM'} at 127.10.191.193:38719
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab.
Entry for principal kudu/127.10.191.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab.
I20260620 23:10:45.260222 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.194:0
--local_ip_for_outbound_sockets=127.10.191.194
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:46343
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:35837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:10:45.568809 11255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:10:45.569078 11255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:10:45.569250 11255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:10:45.576543 11255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:10:45.576648 11255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:10:45.576853 11255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.194
I20260620 23:10:45.586251 11255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:35837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:46343
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.194
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:10:45.587532 11255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:10:45.588796 11255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:10:45.601384 11260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:45.601297 11263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:45.601320 11261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:10:45.602267 11255 server_base.cc:1034] running on GCE node
I20260620 23:10:45.602795 11255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:10:45.603947 11255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:10:45.605191 11255 hybrid_clock.cc:648] HybridClock initialized: now 1781997045605128 us; error 58 us; skew 500 ppm
I20260620 23:10:45.605535 11255 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:10:45 dist-test-slave-jkbq krb5kdc[11014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997045, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:10:45.609062 11255 init.cc:377] Logged in from keytab as kudu/127.10.191.194@KRBTEST.COM (short username kudu)
I20260620 23:10:45.610548 11255 webserver.cc:460] Webserver started at http://127.10.191.194:45143/ using document root <none> and password file <none>
I20260620 23:10:45.611249 11255 fs_manager.cc:359] Metadata directory not provided
I20260620 23:10:45.611361 11255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:10:45.611613 11255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:10:45.614331 11255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-1/data/instance:
uuid: "edf5874bf4074cdcaea2e0165992b31b"
format_stamp: "Formatted at 2026-06-20 23:10:45 on dist-test-slave-jkbq"
server_key: "4cfcb8763d405ffb8f677a6be14ba985"
server_key_iv: "77887834c3e3dd7f46a4084a1e5a8d3d"
server_key_version: "encryptionkey@0"
I20260620 23:10:45.615149 11255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance:
uuid: "edf5874bf4074cdcaea2e0165992b31b"
format_stamp: "Formatted at 2026-06-20 23:10:45 on dist-test-slave-jkbq"
server_key: "4cfcb8763d405ffb8f677a6be14ba985"
server_key_iv: "77887834c3e3dd7f46a4084a1e5a8d3d"
server_key_version: "encryptionkey@0"
I20260620 23:10:45.620296 11255 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 23:10:45.623648 11270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:45.625000 11255 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:10:45.625178 11255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-1/wal
uuid: "edf5874bf4074cdcaea2e0165992b31b"
format_stamp: "Formatted at 2026-06-20 23:10:45 on dist-test-slave-jkbq"
server_key: "4cfcb8763d405ffb8f677a6be14ba985"
server_key_iv: "77887834c3e3dd7f46a4084a1e5a8d3d"
server_key_version: "encryptionkey@0"
I20260620 23:10:45.625387 11255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:10:45.641697 11255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:10:45.646059 11255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:10:45.646360 11255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:10:45.647423 11255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:10:45.648761 11255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:10:45.648831 11255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:45.649019 11255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:10:45.649068 11255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:45.681747 11255 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.194:44697
I20260620 23:10:45.681789 11382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.194:44697 every 8 connection(s)
I20260620 23:10:45.683077 11255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
I20260620 23:10:45.686167 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 11255
I20260620 23:10:45.686344 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance
I20260620 23:10:45.686964 11007 external_mini_cluster.cc:1428] Setting key 66d6925c176a75d1a54d5041cb6183af
Jun 20 23:10:45 dist-test-slave-jkbq krb5kdc[11014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997045, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.191.195@KRBTEST.COM; defaulting to no policy
I20260620 23:10:45.704849 11251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:45.685652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:46565 (local address 127.10.191.254:46343)
0620 23:10:45.685872 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:45.685879 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:45.687152 (+ 1273us) server_negotiation.cc:407] Connection header received
0620 23:10:45.689038 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:45.689044 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:45.689109 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:45.689189 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:45.691379 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:45.692093 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:45.694131 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:45.694696 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:45.694804 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:45.699044 (+ 4240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:45.699049 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:45.699052 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:45.699085 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:45.701680 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:45.702415 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:45.702420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:45.702426 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:45.702557 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:45.703076 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:45.703080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:45.703085 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:45.703362 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:45.703517 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:45.704268 (+ 751us) server_negotiation.cc:299] Negotiation successful
0620 23:10:45.704490 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.195@KRBTEST.COM" created.
I20260620 23:10:45.705968 11385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:45.686102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:46343 (local address 127.10.191.194:46565)
0620 23:10:45.686972 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:45.687032 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:45.688719 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:45.689478 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:45.689488 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:45.690287 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:10:45.691181 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:45.691209 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:45.692252 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:45.692260 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:45.693973 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:45.693985 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:45.694987 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:45.694998 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:45.695330 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:45.696046 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:10:45.696080 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:10:45.698788 (+ 2708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:10:45.701827 (+ 3039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:45.701837 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:45.701854 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:45.702280 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:45.702669 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:45.702675 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:45.702678 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:45.702959 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:45.703488 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:10:45.703496 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:10:45.703920 (+ 424us) client_negotiation.cc:769] Sending connection context
0620 23:10:45.704190 (+ 270us) client_negotiation.cc:240] Negotiation successful
0620 23:10:45.704727 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":359,"threads_started":1}
I20260620 23:10:45.707444 11383 heartbeater.cc:347] Connected to a master server at 127.10.191.254:46343
I20260620 23:10:45.707783 11383 heartbeater.cc:464] Registering TS with master...
I20260620 23:10:45.708711 11383 heartbeater.cc:511] Master 127.10.191.254:46343 requested a full tablet report, sending...
I20260620 23:10:45.710492 11059 ts_manager.cc:194] Registered new tserver with Master: edf5874bf4074cdcaea2e0165992b31b (127.10.191.194:44697)
I20260620 23:10:45.711403 11059 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.194@KRBTEST.COM'} at 127.10.191.194:46565
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab.
Entry for principal kudu/127.10.191.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab.
I20260620 23:10:45.726146 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.195:0
--local_ip_for_outbound_sockets=127.10.191.195
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:46343
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:35837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:10:46.039680 11388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:10:46.039950 11388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:10:46.040045 11388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:10:46.047179 11388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:10:46.047358 11388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:10:46.047526 11388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.195
I20260620 23:10:46.056882 11388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:35837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:46343
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.195
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:10:46.058154 11388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:10:46.059532 11388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:10:46.072515 11393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:46.072535 11394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:46.072467 11396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:10:46.073293 11388 server_base.cc:1034] running on GCE node
I20260620 23:10:46.073884 11388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:10:46.074728 11388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:10:46.075935 11388 hybrid_clock.cc:648] HybridClock initialized: now 1781997046075883 us; error 68 us; skew 500 ppm
I20260620 23:10:46.076198 11388 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:10:46 dist-test-slave-jkbq krb5kdc[11014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997046, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:10:46.079274 11388 init.cc:377] Logged in from keytab as kudu/127.10.191.195@KRBTEST.COM (short username kudu)
I20260620 23:10:46.080842 11388 webserver.cc:460] Webserver started at http://127.10.191.195:43025/ using document root <none> and password file <none>
I20260620 23:10:46.081502 11388 fs_manager.cc:359] Metadata directory not provided
I20260620 23:10:46.081631 11388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:10:46.081951 11388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:10:46.084765 11388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-2/data/instance:
uuid: "4eb5e894f68d4546b4c14aa78d270985"
format_stamp: "Formatted at 2026-06-20 23:10:46 on dist-test-slave-jkbq"
server_key: "a75c6c5d4ea36d838f0a173737603788"
server_key_iv: "3cebcb6548f8c77923420684d18c135e"
server_key_version: "encryptionkey@0"
I20260620 23:10:46.085505 11388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance:
uuid: "4eb5e894f68d4546b4c14aa78d270985"
format_stamp: "Formatted at 2026-06-20 23:10:46 on dist-test-slave-jkbq"
server_key: "a75c6c5d4ea36d838f0a173737603788"
server_key_iv: "3cebcb6548f8c77923420684d18c135e"
server_key_version: "encryptionkey@0"
I20260620 23:10:46.090502 11388 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 23:10:46.093861 11403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:46.095202 11388 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:10:46.095369 11388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-2/wal
uuid: "4eb5e894f68d4546b4c14aa78d270985"
format_stamp: "Formatted at 2026-06-20 23:10:46 on dist-test-slave-jkbq"
server_key: "a75c6c5d4ea36d838f0a173737603788"
server_key_iv: "3cebcb6548f8c77923420684d18c135e"
server_key_version: "encryptionkey@0"
I20260620 23:10:46.095561 11388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:10:46.123986 11388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:10:46.128322 11388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:10:46.128638 11388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:10:46.129658 11388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:10:46.131031 11388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:10:46.131126 11388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:46.131249 11388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:10:46.131337 11388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:46.165427 11388 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.195:45591
I20260620 23:10:46.165475 11515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.195:45591 every 8 connection(s)
I20260620 23:10:46.166694 11388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
I20260620 23:10:46.172108 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 11388
I20260620 23:10:46.172291 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance
I20260620 23:10:46.172818 11007 external_mini_cluster.cc:1428] Setting key 8d764677648947a9a5203d1d1d4a1da2
Jun 20 23:10:46 dist-test-slave-jkbq krb5kdc[11014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997046, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:10:46.186205 11251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:46.169240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:43237 (local address 127.10.191.254:46343)
0620 23:10:46.169409 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:46.169414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:46.170573 (+ 1159us) server_negotiation.cc:407] Connection header received
0620 23:10:46.172735 (+ 2162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:46.172740 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:46.172825 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:46.172947 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:46.174758 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:46.175656 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:46.177364 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:46.177787 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:46.177885 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:46.180972 (+ 3087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:46.180977 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:46.180980 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:46.181011 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:46.183679 (+ 2668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:46.184342 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:46.184346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:46.184348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:46.184459 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:46.184859 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:46.184862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:46.184864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:46.185096 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:46.185203 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:46.185820 (+ 617us) server_negotiation.cc:299] Negotiation successful
0620 23:10:46.185975 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 23:10:46.187244 11518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:46.169755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:46343 (local address 127.10.191.195:43237)
0620 23:10:46.170419 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:46.170459 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:46.172412 (+ 1953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:46.173209 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:46.173221 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:46.173863 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:10:46.174559 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:46.174586 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:46.175800 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:46.175806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:46.177233 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:46.177244 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:46.177973 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:46.177978 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:46.178160 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:46.178745 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:10:46.178765 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:10:46.180800 (+ 2035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:10:46.183824 (+ 3024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:46.183831 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:46.183841 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:46.184237 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:46.184571 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:46.184575 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:46.184577 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:46.184752 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:46.185195 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:10:46.185200 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:10:46.185493 (+ 293us) client_negotiation.cc:769] Sending connection context
0620 23:10:46.185705 (+ 212us) client_negotiation.cc:240] Negotiation successful
0620 23:10:46.186073 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":260,"threads_started":1}
I20260620 23:10:46.188589 11516 heartbeater.cc:347] Connected to a master server at 127.10.191.254:46343
I20260620 23:10:46.188920 11516 heartbeater.cc:464] Registering TS with master...
I20260620 23:10:46.189574 11516 heartbeater.cc:511] Master 127.10.191.254:46343 requested a full tablet report, sending...
I20260620 23:10:46.190981 11059 ts_manager.cc:194] Registered new tserver with Master: 4eb5e894f68d4546b4c14aa78d270985 (127.10.191.195:45591)
I20260620 23:10:46.191769 11059 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.195@KRBTEST.COM'} at 127.10.191.195:43237
I20260620 23:10:46.200950 11007 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 23:10:46 dist-test-slave-jkbq krb5kdc[11014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997046, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 23:10:46 dist-test-slave-jkbq krb5kdc[11014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997046, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.191.193@KRBTEST.COM
I20260620 23:10:46.243440 11522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:46.228825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46028 (local address 127.10.191.193:33107)
0620 23:10:46.229408 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:46.229415 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:46.229522 (+ 107us) server_negotiation.cc:407] Connection header received
0620 23:10:46.229663 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:46.229670 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:46.229954 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:46.230132 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:46.231547 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:46.232549 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:46.233822 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:46.234381 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:46.234490 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:46.238197 (+ 3707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:46.238207 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:46.238220 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:46.238252 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:46.240967 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:46.241561 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:46.241568 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:46.241570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:46.241694 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:46.242044 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:46.242047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:46.242049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:46.242368 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:46.242598 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:46.242944 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 23:10:46.243151 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":292,"threads_started":1}
W20260620 23:10:46.244730 11164 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46028
I20260620 23:10:46.251724 11249 heartbeater.cc:503] Master 127.10.191.254:46343 was elected leader, sending a full tablet report...
Jun 20 23:10:46 dist-test-slave-jkbq krb5kdc[11014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 23:10:46 dist-test-slave-jkbq krb5kdc[11014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.193@KRBTEST.COM
I20260620 23:10:46.288795 11522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:46.273872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46038 (local address 127.10.191.193:33107)
0620 23:10:46.274096 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:46.274103 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:46.274278 (+ 175us) server_negotiation.cc:407] Connection header received
0620 23:10:46.274499 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:46.274503 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:46.274593 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:46.274701 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:46.276050 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:46.276756 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:46.277970 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:46.278380 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:46.278504 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:46.282904 (+ 4400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:46.282911 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:46.282916 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:46.282975 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:46.286096 (+ 3121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:46.286775 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:46.286788 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:46.286792 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:46.287016 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:46.287464 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:46.287469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:46.287473 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:46.287814 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:46.288099 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:46.288401 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 23:10:46.288569 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260620 23:10:46.290235 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 11121
I20260620 23:10:46.303351 11522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:46.291127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46050 (local address 127.10.191.193:33107)
0620 23:10:46.291303 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:46.291310 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:46.291633 (+ 323us) server_negotiation.cc:407] Connection header received
0620 23:10:46.291836 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:46.291843 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:46.291916 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:46.292028 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:46.293525 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:46.294532 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:46.295802 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:46.296229 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:46.296369 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:46.298476 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:46.298482 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:46.298486 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:46.298522 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:46.300948 (+ 2426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:46.301535 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:46.301542 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:46.301545 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:46.301667 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:46.302010 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:46.302016 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:46.302019 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:46.302265 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:46.302490 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:46.302790 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 23:10:46.303007 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 23:10:46.304095 11244 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:10:46.385854 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 11121
I20260620 23:10:46.401641 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 11255
Jun 20 23:10:46 dist-test-slave-jkbq krb5kdc[11014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.194@KRBTEST.COM
I20260620 23:10:46.417361 11530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:46.402576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34854 (local address 127.10.191.194:44697)
0620 23:10:46.403145 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:46.403152 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:46.403175 (+ 23us) server_negotiation.cc:407] Connection header received
0620 23:10:46.403247 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:46.403254 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:46.403464 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:46.403640 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:46.404795 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:46.405808 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:46.407110 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:46.407642 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:46.407780 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:46.411812 (+ 4032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:46.411822 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:46.411836 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:46.411867 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:46.414489 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:46.415283 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:46.415291 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:46.415293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:46.415414 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:46.415826 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:46.415835 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:46.415837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:46.416148 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:46.416446 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:46.416751 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 23:10:46.417010 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":341,"threads_started":1}
I20260620 23:10:46.418780 11378 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:10:46.500507 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 11255
I20260620 23:10:46.516131 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 11388
Jun 20 23:10:46 dist-test-slave-jkbq krb5kdc[11014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.195@KRBTEST.COM
I20260620 23:10:46.531659 11532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:46.517096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60718 (local address 127.10.191.195:45591)
0620 23:10:46.517665 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:46.517672 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:46.517701 (+ 29us) server_negotiation.cc:407] Connection header received
0620 23:10:46.517759 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:46.517765 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:46.517995 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:46.518121 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:46.519455 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:46.520516 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:46.521729 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:46.522285 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:46.522387 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:46.526338 (+ 3951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:46.526349 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:46.526365 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:46.526398 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:46.529188 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:46.529786 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:46.529792 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:46.529794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:46.529906 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:46.530195 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:46.530198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:46.530200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:46.530512 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:46.530730 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:46.531088 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 23:10:46.531337 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":277,"threads_started":1}
I20260620 23:10:46.532986 11511 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:10:46.619141 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 11388
I20260620 23:10:46.634487 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 11027
I20260620 23:10:46.635710 11089 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:10:46.737121 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 11027
2026-06-20T23:10:46Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3811 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:10:46 dist-test-slave-jkbq krb5kdc[11537](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:10:46 dist-test-slave-jkbq krb5kdc[11537](info): set up 2 sockets
Jun 20 23:10:46 dist-test-slave-jkbq krb5kdc[11537](info): commencing operation
krb5kdc: starting...
W20260620 23:10:47.826702 11007 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Jun 20 23:10:47 dist-test-slave-jkbq krb5kdc[11537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:10:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:10:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.191.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
I20260620 23:10:47.999151 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:36357
--webserver_interface=127.10.191.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:36847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:36357
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:10:48.310532 11550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:10:48.310808 11550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:10:48.311009 11550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:10:48.317974 11550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:10:48.318066 11550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:10:48.318172 11550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:10:48.318271 11550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:10:48.318347 11550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:10:48.327780 11550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:36847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:36357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:36357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:10:48.329061 11550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:10:48.330363 11550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:10:48.341765 11555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:48.341864 11556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:48.341883 11558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:10:48.342581 11550 server_base.cc:1034] running on GCE node
I20260620 23:10:48.343422 11550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:10:48.344837 11550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:10:48.346060 11550 hybrid_clock.cc:648] HybridClock initialized: now 1781997048346005 us; error 41 us; skew 500 ppm
I20260620 23:10:48.346362 11550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:10:48 dist-test-slave-jkbq krb5kdc[11537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997048, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:10:48.349436 11550 init.cc:377] Logged in from keytab as kudu/127.10.191.254@KRBTEST.COM (short username kudu)
I20260620 23:10:48.350945 11550 webserver.cc:460] Webserver started at http://127.10.191.254:45389/ using document root <none> and password file <none>
I20260620 23:10:48.351599 11550 fs_manager.cc:359] Metadata directory not provided
I20260620 23:10:48.351773 11550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:10:48.352181 11550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:10:48.355129 11550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "9bd81990bcd342b183da2403094e7abc"
format_stamp: "Formatted at 2026-06-20 23:10:48 on dist-test-slave-jkbq"
server_key: "2d091587f1d1ce99306b4c7444f31a16"
server_key_iv: "9daa6ac5287a42c5661e6785e064d1f7"
server_key_version: "encryptionkey@0"
I20260620 23:10:48.355876 11550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "9bd81990bcd342b183da2403094e7abc"
format_stamp: "Formatted at 2026-06-20 23:10:48 on dist-test-slave-jkbq"
server_key: "2d091587f1d1ce99306b4c7444f31a16"
server_key_iv: "9daa6ac5287a42c5661e6785e064d1f7"
server_key_version: "encryptionkey@0"
I20260620 23:10:48.360823 11550 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:10:48.364140 11565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:48.365420 11550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:10:48.365597 11550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "9bd81990bcd342b183da2403094e7abc"
format_stamp: "Formatted at 2026-06-20 23:10:48 on dist-test-slave-jkbq"
server_key: "2d091587f1d1ce99306b4c7444f31a16"
server_key_iv: "9daa6ac5287a42c5661e6785e064d1f7"
server_key_version: "encryptionkey@0"
I20260620 23:10:48.365806 11550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:10:48.377738 11550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:10:48.381969 11550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:10:48.382249 11550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:10:48.400580 11550 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:36357
I20260620 23:10:48.400604 11616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:36357 every 8 connection(s)
I20260620 23:10:48.401979 11550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:10:48.403757 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 11550
I20260620 23:10:48.404011 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:10:48.404650 11007 external_mini_cluster.cc:1428] Setting key 07233faddbfbe4b31a41665e6ed9303c
I20260620 23:10:48.406219 11617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 23:10:48 dist-test-slave-jkbq krb5kdc[11537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:10:48.419526 11617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bd81990bcd342b183da2403094e7abc: Bootstrap starting.
I20260620 23:10:48.423635 11617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bd81990bcd342b183da2403094e7abc: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:48.424887 11617 log.cc:824] T 00000000000000000000000000000000 P 9bd81990bcd342b183da2403094e7abc: Log is configured to *not* fsync() on all Append() calls
I20260620 23:10:48.428236 11617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bd81990bcd342b183da2403094e7abc: No bootstrap required, opened a new log
I20260620 23:10:48.428551 11620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:48.406580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35934 (local address 127.10.191.254:36357)
0620 23:10:48.407442 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:48.407458 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:48.407508 (+ 50us) server_negotiation.cc:407] Connection header received
0620 23:10:48.409472 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:48.409499 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:48.410027 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:48.410472 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:48.411854 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:48.413883 (+ 2029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:48.415204 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:48.416222 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:48.416411 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:48.420264 (+ 3853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:48.420274 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:48.420294 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:48.420345 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:48.424277 (+ 3932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:48.424927 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:48.424935 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:48.424943 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:48.425135 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:48.425551 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:48.425558 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:48.425561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:48.426105 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:48.426429 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:48.426825 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 23:10:48.427256 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":374,"threads_started":1}
I20260620 23:10:48.436457 11617 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bd81990bcd342b183da2403094e7abc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd81990bcd342b183da2403094e7abc" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 36357 } }
I20260620 23:10:48.436820 11617 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9bd81990bcd342b183da2403094e7abc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:10:48.436936 11617 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bd81990bcd342b183da2403094e7abc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bd81990bcd342b183da2403094e7abc, State: Initialized, Role: FOLLOWER
I20260620 23:10:48.437590 11617 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bd81990bcd342b183da2403094e7abc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd81990bcd342b183da2403094e7abc" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 36357 } }
I20260620 23:10:48.437847 11617 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9bd81990bcd342b183da2403094e7abc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:10:48.438006 11617 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bd81990bcd342b183da2403094e7abc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:10:48.438195 11617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bd81990bcd342b183da2403094e7abc [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:10:48.440359 11617 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bd81990bcd342b183da2403094e7abc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd81990bcd342b183da2403094e7abc" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 36357 } }
I20260620 23:10:48.440826 11617 leader_election.cc:302] T 00000000000000000000000000000000 P 9bd81990bcd342b183da2403094e7abc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bd81990bcd342b183da2403094e7abc; no voters:
I20260620 23:10:48.441601 11617 leader_election.cc:288] T 00000000000000000000000000000000 P 9bd81990bcd342b183da2403094e7abc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:10:48.441718 11622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bd81990bcd342b183da2403094e7abc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:10:48.443074 11622 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bd81990bcd342b183da2403094e7abc [term 1 LEADER]: Becoming Leader. State: Replica: 9bd81990bcd342b183da2403094e7abc, State: Running, Role: LEADER
I20260620 23:10:48.443578 11622 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bd81990bcd342b183da2403094e7abc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd81990bcd342b183da2403094e7abc" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 36357 } }
I20260620 23:10:48.443802 11617 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bd81990bcd342b183da2403094e7abc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:10:48.446636 11623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bd81990bcd342b183da2403094e7abc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bd81990bcd342b183da2403094e7abc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd81990bcd342b183da2403094e7abc" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 36357 } } }
I20260620 23:10:48.446609 11624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bd81990bcd342b183da2403094e7abc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bd81990bcd342b183da2403094e7abc. Latest consensus state: current_term: 1 leader_uuid: "9bd81990bcd342b183da2403094e7abc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd81990bcd342b183da2403094e7abc" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 36357 } } }
I20260620 23:10:48.447182 11623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bd81990bcd342b183da2403094e7abc [sys.catalog]: This master's current role is: LEADER
I20260620 23:10:48.447183 11624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bd81990bcd342b183da2403094e7abc [sys.catalog]: This master's current role is: LEADER
I20260620 23:10:48.448045 11631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:10:48.453557 11631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:10:48.463039 11631 catalog_manager.cc:1347] Generated new cluster ID: ea5fc3435ec640fdb48d1b45a40b2ba8
I20260620 23:10:48.463209 11631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:10:48.485191 11631 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:10:48.487079 11631 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:10:48.494069 11631 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9bd81990bcd342b183da2403094e7abc: Generated new TSK 0
I20260620 23:10:48.494833 11631 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.191.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab.
Entry for principal kudu/127.10.191.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab.
I20260620 23:10:48.542789 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.193:0
--local_ip_for_outbound_sockets=127.10.191.193
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:36357
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:36847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:10:48.883524 11643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:10:48.883821 11643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:10:48.883919 11643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:10:48.891613 11643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:10:48.891718 11643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:10:48.891837 11643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.193
I20260620 23:10:48.901849 11643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:36847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:36357
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.193
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:10:48.903208 11643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:10:48.904513 11643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:10:48.917510 11648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:48.917539 11649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:48.917485 11651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:10:48.918345 11643 server_base.cc:1034] running on GCE node
I20260620 23:10:48.918999 11643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:10:48.919914 11643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:10:48.921113 11643 hybrid_clock.cc:648] HybridClock initialized: now 1781997048921063 us; error 39 us; skew 500 ppm
I20260620 23:10:48.921440 11643 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:10:48 dist-test-slave-jkbq krb5kdc[11537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997048, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:10:48.924561 11643 init.cc:377] Logged in from keytab as kudu/127.10.191.193@KRBTEST.COM (short username kudu)
I20260620 23:10:48.925899 11643 webserver.cc:460] Webserver started at http://127.10.191.193:35977/ using document root <none> and password file <none>
I20260620 23:10:48.926584 11643 fs_manager.cc:359] Metadata directory not provided
I20260620 23:10:48.926717 11643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:10:48.927098 11643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:10:48.929932 11643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-0/data/instance:
uuid: "6007d3a942cb4cb7901de09a6cf9063c"
format_stamp: "Formatted at 2026-06-20 23:10:48 on dist-test-slave-jkbq"
server_key: "72c79bc9993cd4c4bfedf8c484e61781"
server_key_iv: "af96994c05bca3a631f41f34368a10ed"
server_key_version: "encryptionkey@0"
I20260620 23:10:48.930769 11643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance:
uuid: "6007d3a942cb4cb7901de09a6cf9063c"
format_stamp: "Formatted at 2026-06-20 23:10:48 on dist-test-slave-jkbq"
server_key: "72c79bc9993cd4c4bfedf8c484e61781"
server_key_iv: "af96994c05bca3a631f41f34368a10ed"
server_key_version: "encryptionkey@0"
I20260620 23:10:48.935721 11643 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 23:10:48.939162 11658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:48.940490 11643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:10:48.940650 11643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-0/wal
uuid: "6007d3a942cb4cb7901de09a6cf9063c"
format_stamp: "Formatted at 2026-06-20 23:10:48 on dist-test-slave-jkbq"
server_key: "72c79bc9993cd4c4bfedf8c484e61781"
server_key_iv: "af96994c05bca3a631f41f34368a10ed"
server_key_version: "encryptionkey@0"
I20260620 23:10:48.940831 11643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:10:48.958999 11643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:10:48.962720 11643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:10:48.963018 11643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:10:48.963946 11643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:10:48.965204 11643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:10:48.965273 11643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:48.965472 11643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:10:48.965545 11643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:48.997970 11643 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.193:34861
I20260620 23:10:48.998054 11770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.193:34861 every 8 connection(s)
I20260620 23:10:48.999316 11643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
I20260620 23:10:49.000901 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 11643
I20260620 23:10:49.001108 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance
I20260620 23:10:49.001698 11007 external_mini_cluster.cc:1428] Setting key 58edb1e3b316feee95c7d2eeaecc3dab
WARNING: no policy specified for kudu/127.10.191.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.194@KRBTEST.COM" created.
Jun 20 23:10:49 dist-test-slave-jkbq krb5kdc[11537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997048, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:10:49.029837 11773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:49.001991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:37361 (local address 127.10.191.254:36357)
0620 23:10:49.002459 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:49.002467 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:49.003689 (+ 1222us) server_negotiation.cc:407] Connection header received
0620 23:10:49.006542 (+ 2853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:49.006546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:49.006609 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:49.006707 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:49.008857 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:49.010046 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:49.012643 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:49.013033 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:49.013134 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:49.023574 (+ 10440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:49.023579 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:49.023582 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:49.023616 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:49.026757 (+ 3141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:49.027554 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:49.027565 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:49.027567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:49.027699 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:49.028135 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:49.028140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:49.028142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:49.028407 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:49.028560 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:49.029245 (+ 685us) server_negotiation.cc:299] Negotiation successful
0620 23:10:49.029460 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":214,"threads_started":1}
I20260620 23:10:49.030973 11774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:49.002646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:36357 (local address 127.10.191.193:37361)
0620 23:10:49.003497 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:49.003549 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:49.006225 (+ 2676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:49.007014 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:49.007028 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:49.007806 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:10:49.008644 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:49.008671 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:49.010237 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:49.010245 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:49.012497 (+ 2252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:49.012513 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:49.013256 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:49.013266 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:49.013570 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:49.018986 (+ 5416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:10:49.019019 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:10:49.023264 (+ 4245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:10:49.026953 (+ 3689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:49.026961 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:49.026975 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:49.027411 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:49.027814 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:49.027820 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:49.027824 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:49.028004 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:49.028535 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:10:49.028544 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:10:49.028891 (+ 347us) client_negotiation.cc:769] Sending connection context
0620 23:10:49.029157 (+ 266us) client_negotiation.cc:240] Negotiation successful
0620 23:10:49.029693 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":341,"threads_started":1}
I20260620 23:10:49.032567 11771 heartbeater.cc:347] Connected to a master server at 127.10.191.254:36357
I20260620 23:10:49.032877 11771 heartbeater.cc:464] Registering TS with master...
I20260620 23:10:49.033581 11771 heartbeater.cc:511] Master 127.10.191.254:36357 requested a full tablet report, sending...
I20260620 23:10:49.035822 11582 ts_manager.cc:194] Registered new tserver with Master: 6007d3a942cb4cb7901de09a6cf9063c (127.10.191.193:34861)
I20260620 23:10:49.037446 11582 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.193@KRBTEST.COM'} at 127.10.191.193:37361
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab.
Entry for principal kudu/127.10.191.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab.
I20260620 23:10:49.045889 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.194:0
--local_ip_for_outbound_sockets=127.10.191.194
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:36357
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:36847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:10:49.372157 11777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:10:49.372407 11777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:10:49.372572 11777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:10:49.379619 11777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:10:49.379714 11777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:10:49.379926 11777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.194
I20260620 23:10:49.389464 11777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:36847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:36357
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.194
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:10:49.390681 11777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:10:49.391975 11777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:10:49.405776 11783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:49.406250 11785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:49.406608 11782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:10:49.406941 11777 server_base.cc:1034] running on GCE node
I20260620 23:10:49.407572 11777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:10:49.408588 11777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:10:49.409821 11777 hybrid_clock.cc:648] HybridClock initialized: now 1781997049409783 us; error 53 us; skew 500 ppm
I20260620 23:10:49.410156 11777 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:10:49 dist-test-slave-jkbq krb5kdc[11537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997049, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:10:49.413579 11777 init.cc:377] Logged in from keytab as kudu/127.10.191.194@KRBTEST.COM (short username kudu)
I20260620 23:10:49.419520 11777 webserver.cc:460] Webserver started at http://127.10.191.194:34147/ using document root <none> and password file <none>
I20260620 23:10:49.420209 11777 fs_manager.cc:359] Metadata directory not provided
I20260620 23:10:49.420354 11777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:10:49.420655 11777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:10:49.423501 11777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-1/data/instance:
uuid: "a4727930eab64257a34ae4f8c4921953"
format_stamp: "Formatted at 2026-06-20 23:10:49 on dist-test-slave-jkbq"
server_key: "296858761dd69db1aec4ac58380d844c"
server_key_iv: "3238bcc983c3688a2b8ae7c555cdad95"
server_key_version: "encryptionkey@0"
I20260620 23:10:49.424280 11777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance:
uuid: "a4727930eab64257a34ae4f8c4921953"
format_stamp: "Formatted at 2026-06-20 23:10:49 on dist-test-slave-jkbq"
server_key: "296858761dd69db1aec4ac58380d844c"
server_key_iv: "3238bcc983c3688a2b8ae7c555cdad95"
server_key_version: "encryptionkey@0"
I20260620 23:10:49.429203 11777 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:10:49.432657 11792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:49.434024 11777 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:10:49.434207 11777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-1/wal
uuid: "a4727930eab64257a34ae4f8c4921953"
format_stamp: "Formatted at 2026-06-20 23:10:49 on dist-test-slave-jkbq"
server_key: "296858761dd69db1aec4ac58380d844c"
server_key_iv: "3238bcc983c3688a2b8ae7c555cdad95"
server_key_version: "encryptionkey@0"
I20260620 23:10:49.434413 11777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:10:49.452663 11777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:10:49.456936 11777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:10:49.457214 11777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:10:49.458163 11777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:10:49.459491 11777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:10:49.459592 11777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:49.459746 11777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:10:49.459820 11777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:49.491393 11777 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.194:36059
I20260620 23:10:49.491446 11904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.194:36059 every 8 connection(s)
I20260620 23:10:49.492632 11777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
I20260620 23:10:49.501472 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 11777
I20260620 23:10:49.501663 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance
I20260620 23:10:49.502079 11007 external_mini_cluster.cc:1428] Setting key 0342725c37fcb79b84ee86721227ae66
Jun 20 23:10:49 dist-test-slave-jkbq krb5kdc[11537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997049, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:10:49.511986 11773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:49.495373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:60247 (local address 127.10.191.254:36357)
0620 23:10:49.495554 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:49.495558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:49.496674 (+ 1116us) server_negotiation.cc:407] Connection header received
0620 23:10:49.498510 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:49.498515 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:49.498603 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:49.498710 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:49.500159 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:49.500926 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:49.502559 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:49.503185 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:49.503326 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:49.506581 (+ 3255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:49.506587 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:49.506590 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:49.506620 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:49.509114 (+ 2494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:49.509789 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:49.509794 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:49.509799 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:49.509925 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:49.510329 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:49.510333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:49.510337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:49.510588 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:49.510739 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:49.511432 (+ 693us) server_negotiation.cc:299] Negotiation successful
0620 23:10:49.511638 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 23:10:49.513161 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:49.495862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:36357 (local address 127.10.191.194:60247)
0620 23:10:49.496511 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:49.496553 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:49.498254 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:49.498980 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:49.498988 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:49.499462 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:10:49.500000 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:49.500016 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:49.501062 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:49.501068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:49.502420 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:49.502431 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:49.503399 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:49.503407 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:49.503682 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:49.504355 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:10:49.504380 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:10:49.506370 (+ 1990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:10:49.509247 (+ 2877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:49.509256 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:49.509270 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:49.509662 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:49.510030 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:49.510036 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:49.510039 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:49.510214 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:49.510712 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:10:49.510719 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:10:49.511097 (+ 378us) client_negotiation.cc:769] Sending connection context
0620 23:10:49.511353 (+ 256us) client_negotiation.cc:240] Negotiation successful
0620 23:10:49.511840 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":257,"threads_started":1}
I20260620 23:10:49.514629 11905 heartbeater.cc:347] Connected to a master server at 127.10.191.254:36357
I20260620 23:10:49.514969 11905 heartbeater.cc:464] Registering TS with master...
I20260620 23:10:49.515842 11905 heartbeater.cc:511] Master 127.10.191.254:36357 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.191.195@KRBTEST.COM; defaulting to no policy
I20260620 23:10:49.517442 11582 ts_manager.cc:194] Registered new tserver with Master: a4727930eab64257a34ae4f8c4921953 (127.10.191.194:36059)
I20260620 23:10:49.518327 11582 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.194@KRBTEST.COM'} at 127.10.191.194:60247
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab.
Entry for principal kudu/127.10.191.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab.
I20260620 23:10:49.538087 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.195:0
--local_ip_for_outbound_sockets=127.10.191.195
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:36357
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:36847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:10:49.849494 11910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:10:49.849766 11910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:10:49.849865 11910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:10:49.856873 11910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:10:49.856984 11910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:10:49.857095 11910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.195
I20260620 23:10:49.866396 11910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:36847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:36357
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.195
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:10:49.867597 11910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:10:49.868839 11910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:10:49.881553 11915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:49.881604 11918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:49.881573 11916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:10:49.882414 11910 server_base.cc:1034] running on GCE node
I20260620 23:10:49.883123 11910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:10:49.884039 11910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:10:49.885237 11910 hybrid_clock.cc:648] HybridClock initialized: now 1781997049885185 us; error 48 us; skew 500 ppm
I20260620 23:10:49.885556 11910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:10:49 dist-test-slave-jkbq krb5kdc[11537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997049, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:10:49.893409 11910 init.cc:377] Logged in from keytab as kudu/127.10.191.195@KRBTEST.COM (short username kudu)
I20260620 23:10:49.895174 11910 webserver.cc:460] Webserver started at http://127.10.191.195:42923/ using document root <none> and password file <none>
I20260620 23:10:49.895874 11910 fs_manager.cc:359] Metadata directory not provided
I20260620 23:10:49.896009 11910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:10:49.896327 11910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:10:49.899231 11910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-2/data/instance:
uuid: "95c280fdf289481492309ddfbf7624cb"
format_stamp: "Formatted at 2026-06-20 23:10:49 on dist-test-slave-jkbq"
server_key: "d9a2fd40f9e3b1df1b9e0076e7bf2146"
server_key_iv: "92a1fc32e1da1b4858b5d3c6e08f8866"
server_key_version: "encryptionkey@0"
I20260620 23:10:49.900012 11910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance:
uuid: "95c280fdf289481492309ddfbf7624cb"
format_stamp: "Formatted at 2026-06-20 23:10:49 on dist-test-slave-jkbq"
server_key: "d9a2fd40f9e3b1df1b9e0076e7bf2146"
server_key_iv: "92a1fc32e1da1b4858b5d3c6e08f8866"
server_key_version: "encryptionkey@0"
I20260620 23:10:49.905057 11910 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:10:49.908468 11925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:49.909718 11910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:10:49.909893 11910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-2/wal
uuid: "95c280fdf289481492309ddfbf7624cb"
format_stamp: "Formatted at 2026-06-20 23:10:49 on dist-test-slave-jkbq"
server_key: "d9a2fd40f9e3b1df1b9e0076e7bf2146"
server_key_iv: "92a1fc32e1da1b4858b5d3c6e08f8866"
server_key_version: "encryptionkey@0"
I20260620 23:10:49.910101 11910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:10:49.922345 11910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:10:49.926352 11910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:10:49.926643 11910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:10:49.927727 11910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:10:49.929090 11910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:10:49.929190 11910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:49.929342 11910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:10:49.929415 11910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:49.963059 11910 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.195:43711
I20260620 23:10:49.963091 12037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.195:43711 every 8 connection(s)
I20260620 23:10:49.964356 11910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
I20260620 23:10:49.973508 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 11910
I20260620 23:10:49.973716 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance
I20260620 23:10:49.974188 11007 external_mini_cluster.cc:1428] Setting key f388d76ad3c99bf531b42a5ccd950b6c
Jun 20 23:10:49 dist-test-slave-jkbq krb5kdc[11537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997049, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:10:49.984854 11773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:49.967070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:36529 (local address 127.10.191.254:36357)
0620 23:10:49.967267 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:49.967272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:49.968512 (+ 1240us) server_negotiation.cc:407] Connection header received
0620 23:10:49.970569 (+ 2057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:49.970577 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:49.970667 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:49.970810 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:49.972403 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:49.973666 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:49.975398 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:49.975799 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:49.975936 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:49.979489 (+ 3553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:49.979493 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:49.979496 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:49.979527 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:49.981880 (+ 2353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:49.982560 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:49.982572 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:49.982574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:49.982696 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:49.983197 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:49.983208 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:49.983211 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:49.983475 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:49.983638 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:49.984280 (+ 642us) server_negotiation.cc:299] Negotiation successful
0620 23:10:49.984499 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 23:10:49.985754 12040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:49.967567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:36357 (local address 127.10.191.195:36529)
0620 23:10:49.968328 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:49.968377 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:49.970251 (+ 1874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:49.971070 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:49.971078 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:49.971629 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:10:49.972216 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:49.972234 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:49.973861 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:49.973867 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:49.975239 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:49.975255 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:49.976022 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:49.976030 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:49.976264 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:49.976979 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:10:49.977012 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:10:49.979275 (+ 2263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:10:49.982014 (+ 2739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:49.982023 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:49.982037 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:49.982437 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:49.982815 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:49.982821 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:49.982824 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:49.983079 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:49.983598 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:10:49.983607 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:10:49.983941 (+ 334us) client_negotiation.cc:769] Sending connection context
0620 23:10:49.984192 (+ 251us) client_negotiation.cc:240] Negotiation successful
0620 23:10:49.984685 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":303,"threads_started":1}
I20260620 23:10:49.987114 12038 heartbeater.cc:347] Connected to a master server at 127.10.191.254:36357
I20260620 23:10:49.987429 12038 heartbeater.cc:464] Registering TS with master...
I20260620 23:10:49.988166 12038 heartbeater.cc:511] Master 127.10.191.254:36357 requested a full tablet report, sending...
I20260620 23:10:49.989651 11581 ts_manager.cc:194] Registered new tserver with Master: 95c280fdf289481492309ddfbf7624cb (127.10.191.195:43711)
I20260620 23:10:49.990513 11581 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.195@KRBTEST.COM'} at 127.10.191.195:36529
I20260620 23:10:50.001782 11007 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:10:50.019650 11773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:50.007731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35938 (local address 127.10.191.254:36357)
0620 23:10:50.007889 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:50.007893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:50.008377 (+ 484us) server_negotiation.cc:407] Connection header received
0620 23:10:50.008567 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:50.008579 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:50.008643 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:50.008744 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:50.010124 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:50.010825 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:50.012184 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:50.012563 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:50.012660 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:50.015194 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:50.015198 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:50.015201 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:50.015230 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:50.017526 (+ 2296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:50.018112 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:50.018116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:50.018118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:50.018233 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:50.018521 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:50.018523 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:50.018525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:50.018806 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:50.019044 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:50.019270 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 23:10:50.019418 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 23:10:50.025856 11581 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35938:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:10:50.032462 11581 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:10:50.040510 11771 heartbeater.cc:503] Master 127.10.191.254:36357 was elected leader, sending a full tablet report...
I20260620 23:10:50.053185 12051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:50.044468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44426 (local address 127.10.191.195:43711)
0620 23:10:50.045188 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:50.045197 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:50.045231 (+ 34us) server_negotiation.cc:407] Connection header received
0620 23:10:50.045333 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:50.045343 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:50.045592 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:50.046526 (+ 934us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:10:50.046725 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:50.048352 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:50.050800 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:50.051991 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:50.052190 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:50.052360 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:50.052681 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 23:10:50.052849 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":2142,"threads_started":1}
I20260620 23:10:50.053277 12050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:50.044136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.195:43711 (local address 127.0.0.1:44426)
0620 23:10:50.044792 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:50.044832 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:50.045070 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:50.045870 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:50.045878 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:50.045913 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:10:50.046362 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:50.046371 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:50.048493 (+ 2122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:50.048498 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:50.050572 (+ 2074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:50.050609 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:50.052192 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:50.052198 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:50.052420 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:50.052478 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 23:10:50.052631 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 23:10:50.052812 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":3,"thread_start_us":254,"threads_started":1}
I20260620 23:10:50.054919 12052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:50.045174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:34861 (local address 127.0.0.1:48062)
0620 23:10:50.046015 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:50.046037 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:50.046134 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:50.047371 (+ 1237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:50.047378 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:50.047402 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:10:50.047854 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:50.047866 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:50.050157 (+ 2291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:50.050163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:50.052440 (+ 2277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:50.052454 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:50.054124 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:50.054130 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:50.054292 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:50.054328 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 23:10:50.054424 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 23:10:50.054553 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"mutex_wait_us":3,"thread_start_us":280,"threads_started":1}
I20260620 23:10:50.055085 12055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:50.046040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48062 (local address 127.10.191.193:34861)
0620 23:10:50.046610 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:50.046620 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:50.046650 (+ 30us) server_negotiation.cc:407] Connection header received
0620 23:10:50.046733 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:50.046740 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:50.047051 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:50.047378 (+ 327us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:10:50.048479 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:50.050012 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:50.052593 (+ 2581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:50.053916 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:50.054124 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:50.054284 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:50.054470 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 23:10:50.054633 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":392,"threads_started":1}
I20260620 23:10:50.057077 11973 tablet_service.cc:1467] Processing CreateTablet for tablet 811dd0ca1de24596b1d379978c9d73ad (DEFAULT_TABLE table=test-table [id=0e8f33a7838149a6a6a1499b37462eff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:10:50.058259 12053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:50.044989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.194:36059 (local address 127.0.0.1:43604)
0620 23:10:50.045664 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:50.045682 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:50.045801 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:50.048879 (+ 3078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:50.048883 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:50.048907 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:10:50.049362 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:50.049371 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:50.050977 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:50.050987 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:50.053127 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:50.053140 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:50.055817 (+ 2677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:50.055824 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:50.055989 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:50.056022 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 23:10:50.057765 (+ 1743us) client_negotiation.cc:240] Negotiation successful
0620 23:10:50.057884 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":27,"thread_start_us":1242,"threads_started":1}
I20260620 23:10:50.058005 11706 tablet_service.cc:1467] Processing CreateTablet for tablet 811dd0ca1de24596b1d379978c9d73ad (DEFAULT_TABLE table=test-table [id=0e8f33a7838149a6a6a1499b37462eff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:10:50.058463 11973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 811dd0ca1de24596b1d379978c9d73ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:50.059180 12054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:50.045556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43604 (local address 127.10.191.194:36059)
0620 23:10:50.048377 (+ 2821us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:50.048387 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:50.048415 (+ 28us) server_negotiation.cc:407] Connection header received
0620 23:10:50.048504 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:50.048513 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:50.048705 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:50.048888 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:10:50.049516 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:50.050794 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:50.053756 (+ 2962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:50.055526 (+ 1770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:50.058488 (+ 2962us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:50.058637 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:50.058696 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 23:10:50.058854 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2670,"thread_start_us":2572,"threads_started":1}
I20260620 23:10:50.059249 11706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 811dd0ca1de24596b1d379978c9d73ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:50.062453 11840 tablet_service.cc:1467] Processing CreateTablet for tablet 811dd0ca1de24596b1d379978c9d73ad (DEFAULT_TABLE table=test-table [id=0e8f33a7838149a6a6a1499b37462eff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:10:50.063776 11840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 811dd0ca1de24596b1d379978c9d73ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:50.070173 12056 tablet_bootstrap.cc:492] T 811dd0ca1de24596b1d379978c9d73ad P 6007d3a942cb4cb7901de09a6cf9063c: Bootstrap starting.
I20260620 23:10:50.073585 12056 tablet_bootstrap.cc:654] T 811dd0ca1de24596b1d379978c9d73ad P 6007d3a942cb4cb7901de09a6cf9063c: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:50.074421 12057 tablet_bootstrap.cc:492] T 811dd0ca1de24596b1d379978c9d73ad P 95c280fdf289481492309ddfbf7624cb: Bootstrap starting.
I20260620 23:10:50.074944 12056 log.cc:824] T 811dd0ca1de24596b1d379978c9d73ad P 6007d3a942cb4cb7901de09a6cf9063c: Log is configured to *not* fsync() on all Append() calls
I20260620 23:10:50.077620 12059 tablet_bootstrap.cc:492] T 811dd0ca1de24596b1d379978c9d73ad P a4727930eab64257a34ae4f8c4921953: Bootstrap starting.
I20260620 23:10:50.079510 12057 tablet_bootstrap.cc:654] T 811dd0ca1de24596b1d379978c9d73ad P 95c280fdf289481492309ddfbf7624cb: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:50.080546 12059 tablet_bootstrap.cc:654] T 811dd0ca1de24596b1d379978c9d73ad P a4727930eab64257a34ae4f8c4921953: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:50.080750 12056 tablet_bootstrap.cc:492] T 811dd0ca1de24596b1d379978c9d73ad P 6007d3a942cb4cb7901de09a6cf9063c: No bootstrap required, opened a new log
I20260620 23:10:50.080753 12057 log.cc:824] T 811dd0ca1de24596b1d379978c9d73ad P 95c280fdf289481492309ddfbf7624cb: Log is configured to *not* fsync() on all Append() calls
I20260620 23:10:50.081019 12056 ts_tablet_manager.cc:1397] T 811dd0ca1de24596b1d379978c9d73ad P 6007d3a942cb4cb7901de09a6cf9063c: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260620 23:10:50.081712 12059 log.cc:824] T 811dd0ca1de24596b1d379978c9d73ad P a4727930eab64257a34ae4f8c4921953: Log is configured to *not* fsync() on all Append() calls
I20260620 23:10:50.083642 12057 tablet_bootstrap.cc:492] T 811dd0ca1de24596b1d379978c9d73ad P 95c280fdf289481492309ddfbf7624cb: No bootstrap required, opened a new log
I20260620 23:10:50.083902 12057 ts_tablet_manager.cc:1397] T 811dd0ca1de24596b1d379978c9d73ad P 95c280fdf289481492309ddfbf7624cb: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260620 23:10:50.084167 12059 tablet_bootstrap.cc:492] T 811dd0ca1de24596b1d379978c9d73ad P a4727930eab64257a34ae4f8c4921953: No bootstrap required, opened a new log
I20260620 23:10:50.084399 12059 ts_tablet_manager.cc:1397] T 811dd0ca1de24596b1d379978c9d73ad P a4727930eab64257a34ae4f8c4921953: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 23:10:50.091279 12059 raft_consensus.cc:348] T 811dd0ca1de24596b1d379978c9d73ad P a4727930eab64257a34ae4f8c4921953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95c280fdf289481492309ddfbf7624cb" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 43711 } } peers { permanent_uuid: "6007d3a942cb4cb7901de09a6cf9063c" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 34861 } } peers { permanent_uuid: "a4727930eab64257a34ae4f8c4921953" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 36059 } }
I20260620 23:10:50.091357 12056 raft_consensus.cc:348] T 811dd0ca1de24596b1d379978c9d73ad P 6007d3a942cb4cb7901de09a6cf9063c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95c280fdf289481492309ddfbf7624cb" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 43711 } } peers { permanent_uuid: "6007d3a942cb4cb7901de09a6cf9063c" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 34861 } } peers { permanent_uuid: "a4727930eab64257a34ae4f8c4921953" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 36059 } }
I20260620 23:10:50.091639 12059 raft_consensus.cc:374] T 811dd0ca1de24596b1d379978c9d73ad P a4727930eab64257a34ae4f8c4921953 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:10:50.091744 12059 raft_consensus.cc:729] T 811dd0ca1de24596b1d379978c9d73ad P a4727930eab64257a34ae4f8c4921953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4727930eab64257a34ae4f8c4921953, State: Initialized, Role: FOLLOWER
I20260620 23:10:50.091763 12056 raft_consensus.cc:374] T 811dd0ca1de24596b1d379978c9d73ad P 6007d3a942cb4cb7901de09a6cf9063c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:10:50.091857 12056 raft_consensus.cc:729] T 811dd0ca1de24596b1d379978c9d73ad P 6007d3a942cb4cb7901de09a6cf9063c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6007d3a942cb4cb7901de09a6cf9063c, State: Initialized, Role: FOLLOWER
I20260620 23:10:50.092342 12059 consensus_queue.cc:260] T 811dd0ca1de24596b1d379978c9d73ad P a4727930eab64257a34ae4f8c4921953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95c280fdf289481492309ddfbf7624cb" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 43711 } } peers { permanent_uuid: "6007d3a942cb4cb7901de09a6cf9063c" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 34861 } } peers { permanent_uuid: "a4727930eab64257a34ae4f8c4921953" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 36059 } }
I20260620 23:10:50.092494 12056 consensus_queue.cc:260] T 811dd0ca1de24596b1d379978c9d73ad P 6007d3a942cb4cb7901de09a6cf9063c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95c280fdf289481492309ddfbf7624cb" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 43711 } } peers { permanent_uuid: "6007d3a942cb4cb7901de09a6cf9063c" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 34861 } } peers { permanent_uuid: "a4727930eab64257a34ae4f8c4921953" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 36059 } }
I20260620 23:10:50.093510 11905 heartbeater.cc:503] Master 127.10.191.254:36357 was elected leader, sending a full tablet report...
I20260620 23:10:50.093964 12059 ts_tablet_manager.cc:1428] T 811dd0ca1de24596b1d379978c9d73ad P a4727930eab64257a34ae4f8c4921953: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
I20260620 23:10:50.095357 12057 raft_consensus.cc:348] T 811dd0ca1de24596b1d379978c9d73ad P 95c280fdf289481492309ddfbf7624cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95c280fdf289481492309ddfbf7624cb" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 43711 } } peers { permanent_uuid: "6007d3a942cb4cb7901de09a6cf9063c" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 34861 } } peers { permanent_uuid: "a4727930eab64257a34ae4f8c4921953" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 36059 } }
I20260620 23:10:50.095721 12057 raft_consensus.cc:374] T 811dd0ca1de24596b1d379978c9d73ad P 95c280fdf289481492309ddfbf7624cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:10:50.095834 12057 raft_consensus.cc:729] T 811dd0ca1de24596b1d379978c9d73ad P 95c280fdf289481492309ddfbf7624cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95c280fdf289481492309ddfbf7624cb, State: Initialized, Role: FOLLOWER
I20260620 23:10:50.096356 12057 consensus_queue.cc:260] T 811dd0ca1de24596b1d379978c9d73ad P 95c280fdf289481492309ddfbf7624cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95c280fdf289481492309ddfbf7624cb" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 43711 } } peers { permanent_uuid: "6007d3a942cb4cb7901de09a6cf9063c" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 34861 } } peers { permanent_uuid: "a4727930eab64257a34ae4f8c4921953" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 36059 } }
I20260620 23:10:50.097636 12056 ts_tablet_manager.cc:1428] T 811dd0ca1de24596b1d379978c9d73ad P 6007d3a942cb4cb7901de09a6cf9063c: Time spent starting tablet: real 0.016s user 0.017s sys 0.000s
I20260620 23:10:50.099534 12057 ts_tablet_manager.cc:1428] T 811dd0ca1de24596b1d379978c9d73ad P 95c280fdf289481492309ddfbf7624cb: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260620 23:10:50.099826 12038 heartbeater.cc:503] Master 127.10.191.254:36357 was elected leader, sending a full tablet report...
W20260620 23:10:50.216001 12039 tablet.cc:2369] T 811dd0ca1de24596b1d379978c9d73ad P 95c280fdf289481492309ddfbf7624cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:10:50.244773 11906 tablet.cc:2369] T 811dd0ca1de24596b1d379978c9d73ad P a4727930eab64257a34ae4f8c4921953: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:10:50.251441 11772 tablet.cc:2369] T 811dd0ca1de24596b1d379978c9d73ad P 6007d3a942cb4cb7901de09a6cf9063c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:10:50.292194 12062 raft_consensus.cc:482] T 811dd0ca1de24596b1d379978c9d73ad P a4727930eab64257a34ae4f8c4921953 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:10:50.292522 12062 raft_consensus.cc:504] T 811dd0ca1de24596b1d379978c9d73ad P a4727930eab64257a34ae4f8c4921953 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95c280fdf289481492309ddfbf7624cb" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 43711 } } peers { permanent_uuid: "6007d3a942cb4cb7901de09a6cf9063c" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 34861 } } peers { permanent_uuid: "a4727930eab64257a34ae4f8c4921953" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 36059 } }
I20260620 23:10:50.294267 12062 leader_election.cc:288] T 811dd0ca1de24596b1d379978c9d73ad P a4727930eab64257a34ae4f8c4921953 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95c280fdf289481492309ddfbf7624cb (127.10.191.195:43711), 6007d3a942cb4cb7901de09a6cf9063c (127.10.191.193:34861)
I20260620 23:10:50.302300 12051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:50.294705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:35131 (local address 127.10.191.195:43711)
0620 23:10:50.294955 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:50.294965 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:50.295395 (+ 430us) server_negotiation.cc:407] Connection header received
0620 23:10:50.295676 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:50.295685 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:50.295787 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:50.295913 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:10:50.296592 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:50.297673 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:50.300142 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:50.301335 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:50.301491 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:50.301576 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:50.301916 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 23:10:50.302057 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 23:10:50.302346 12055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:50.295671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:60391 (local address 127.10.191.193:34861)
0620 23:10:50.295868 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:50.295876 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:50.296319 (+ 443us) server_negotiation.cc:407] Connection header received
0620 23:10:50.296486 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:50.296490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:50.296575 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:50.296671 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:10:50.297103 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:50.298027 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:50.300290 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:50.301501 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:50.301645 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:50.301691 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:50.302024 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 23:10:50.302138 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 23:10:50.302513 12065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:50.294627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.195:43711 (local address 127.10.191.194:35131)
0620 23:10:50.295293 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:50.295308 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:50.295499 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:50.295974 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:50.295981 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:50.296014 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:10:50.296455 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:50.296467 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:50.297792 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:50.297795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:50.299997 (+ 2202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:50.300012 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:50.301535 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:50.301539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:50.301754 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:50.301794 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 23:10:50.301886 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 23:10:50.302061 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"mutex_wait_us":2,"thread_start_us":398,"threads_started":1}
I20260620 23:10:50.302728 12066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:50.295468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:34861 (local address 127.10.191.194:60391)
0620 23:10:50.296214 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:50.296230 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:50.296364 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:50.296683 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:50.296686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:50.296704 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:10:50.296950 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:50.296957 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:50.298137 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:50.298143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:50.300163 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:50.300172 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:50.301691 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:50.301697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:50.301869 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:50.301902 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 23:10:50.301984 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 23:10:50.302114 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"mutex_wait_us":24,"thread_start_us":411,"threads_started":1}
I20260620 23:10:50.303705 11993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "811dd0ca1de24596b1d379978c9d73ad" candidate_uuid: "a4727930eab64257a34ae4f8c4921953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95c280fdf289481492309ddfbf7624cb" is_pre_election: true
I20260620 23:10:50.303705 11726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "811dd0ca1de24596b1d379978c9d73ad" candidate_uuid: "a4727930eab64257a34ae4f8c4921953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6007d3a942cb4cb7901de09a6cf9063c" is_pre_election: true
I20260620 23:10:50.304294 11993 raft_consensus.cc:2454] T 811dd0ca1de24596b1d379978c9d73ad P 95c280fdf289481492309ddfbf7624cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4727930eab64257a34ae4f8c4921953 in term 0.
I20260620 23:10:50.304294 11726 raft_consensus.cc:2454] T 811dd0ca1de24596b1d379978c9d73ad P 6007d3a942cb4cb7901de09a6cf9063c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4727930eab64257a34ae4f8c4921953 in term 0.
I20260620 23:10:50.305112 11793 leader_election.cc:302] T 811dd0ca1de24596b1d379978c9d73ad P a4727930eab64257a34ae4f8c4921953 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95c280fdf289481492309ddfbf7624cb, a4727930eab64257a34ae4f8c4921953; no voters:
I20260620 23:10:50.305615 12062 raft_consensus.cc:2781] T 811dd0ca1de24596b1d379978c9d73ad P a4727930eab64257a34ae4f8c4921953 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:10:50.305833 12062 raft_consensus.cc:482] T 811dd0ca1de24596b1d379978c9d73ad P a4727930eab64257a34ae4f8c4921953 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:10:50.305949 12062 raft_consensus.cc:3037] T 811dd0ca1de24596b1d379978c9d73ad P a4727930eab64257a34ae4f8c4921953 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:10:50.308583 12062 raft_consensus.cc:504] T 811dd0ca1de24596b1d379978c9d73ad P a4727930eab64257a34ae4f8c4921953 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95c280fdf289481492309ddfbf7624cb" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 43711 } } peers { permanent_uuid: "6007d3a942cb4cb7901de09a6cf9063c" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 34861 } } peers { permanent_uuid: "a4727930eab64257a34ae4f8c4921953" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 36059 } }
I20260620 23:10:50.309091 12062 leader_election.cc:288] T 811dd0ca1de24596b1d379978c9d73ad P a4727930eab64257a34ae4f8c4921953 [CANDIDATE]: Term 1 election: Requested vote from peers 95c280fdf289481492309ddfbf7624cb (127.10.191.195:43711), 6007d3a942cb4cb7901de09a6cf9063c (127.10.191.193:34861)
I20260620 23:10:50.309520 11993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "811dd0ca1de24596b1d379978c9d73ad" candidate_uuid: "a4727930eab64257a34ae4f8c4921953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95c280fdf289481492309ddfbf7624cb"
I20260620 23:10:50.309727 11993 raft_consensus.cc:3037] T 811dd0ca1de24596b1d379978c9d73ad P 95c280fdf289481492309ddfbf7624cb [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:10:50.309710 11726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "811dd0ca1de24596b1d379978c9d73ad" candidate_uuid: "a4727930eab64257a34ae4f8c4921953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6007d3a942cb4cb7901de09a6cf9063c"
I20260620 23:10:50.309913 11726 raft_consensus.cc:3037] T 811dd0ca1de24596b1d379978c9d73ad P 6007d3a942cb4cb7901de09a6cf9063c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:10:50.312084 11726 raft_consensus.cc:2454] T 811dd0ca1de24596b1d379978c9d73ad P 6007d3a942cb4cb7901de09a6cf9063c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4727930eab64257a34ae4f8c4921953 in term 1.
I20260620 23:10:50.312104 11993 raft_consensus.cc:2454] T 811dd0ca1de24596b1d379978c9d73ad P 95c280fdf289481492309ddfbf7624cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4727930eab64257a34ae4f8c4921953 in term 1.
I20260620 23:10:50.312450 11793 leader_election.cc:302] T 811dd0ca1de24596b1d379978c9d73ad P a4727930eab64257a34ae4f8c4921953 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6007d3a942cb4cb7901de09a6cf9063c, a4727930eab64257a34ae4f8c4921953; no voters:
I20260620 23:10:50.312769 12062 raft_consensus.cc:2781] T 811dd0ca1de24596b1d379978c9d73ad P a4727930eab64257a34ae4f8c4921953 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:10:50.313179 12062 raft_consensus.cc:686] T 811dd0ca1de24596b1d379978c9d73ad P a4727930eab64257a34ae4f8c4921953 [term 1 LEADER]: Becoming Leader. State: Replica: a4727930eab64257a34ae4f8c4921953, State: Running, Role: LEADER
I20260620 23:10:50.313616 12062 consensus_queue.cc:237] T 811dd0ca1de24596b1d379978c9d73ad P a4727930eab64257a34ae4f8c4921953 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95c280fdf289481492309ddfbf7624cb" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 43711 } } peers { permanent_uuid: "6007d3a942cb4cb7901de09a6cf9063c" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 34861 } } peers { permanent_uuid: "a4727930eab64257a34ae4f8c4921953" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 36059 } }
I20260620 23:10:50.317910 11581 catalog_manager.cc:5495] T 811dd0ca1de24596b1d379978c9d73ad P a4727930eab64257a34ae4f8c4921953 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4727930eab64257a34ae4f8c4921953 (127.10.191.194). New cstate: current_term: 1 leader_uuid: "a4727930eab64257a34ae4f8c4921953" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95c280fdf289481492309ddfbf7624cb" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 43711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6007d3a942cb4cb7901de09a6cf9063c" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 34861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4727930eab64257a34ae4f8c4921953" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 36059 } health_report { overall_health: HEALTHY } } }
Jun 20 23:10:50 dist-test-slave-jkbq krb5kdc[11537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.193@KRBTEST.COM
I20260620 23:10:50.357529 12055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:50.343622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48064 (local address 127.10.191.193:34861)
0620 23:10:50.343814 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:50.343848 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:50.344214 (+ 366us) server_negotiation.cc:407] Connection header received
0620 23:10:50.344444 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:50.344450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:50.344561 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:50.344690 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:50.346101 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:50.346947 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:50.348285 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:50.348766 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:50.348906 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:50.352688 (+ 3782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:50.352701 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:50.352716 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:50.352745 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:50.355303 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:50.355856 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:50.355863 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:50.355865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:50.355981 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:50.356261 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:50.356264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:50.356266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:50.356576 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:50.356783 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:50.357010 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 23:10:50.357156 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 23:10:50 dist-test-slave-jkbq krb5kdc[11537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997050, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 23:10:50 dist-test-slave-jkbq krb5kdc[11537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997050, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.191.193@KRBTEST.COM
I20260620 23:10:50.401197 12055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:50.387513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48072 (local address 127.10.191.193:34861)
0620 23:10:50.387757 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:50.387764 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:50.388063 (+ 299us) server_negotiation.cc:407] Connection header received
0620 23:10:50.388205 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:50.388209 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:50.388274 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:50.388361 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:50.389575 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:50.390303 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:50.391555 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:50.391944 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:50.392036 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:50.395615 (+ 3579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:50.395620 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:50.395623 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:50.395655 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:50.398653 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:50.399231 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:50.399235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:50.399238 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:50.399359 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:50.399692 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:50.399695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:50.399697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:50.399973 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:50.400239 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:50.400585 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 23:10:50.400812 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
Jun 20 23:10:50 dist-test-slave-jkbq krb5kdc[11537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 23:10:50 dist-test-slave-jkbq krb5kdc[11537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.193@KRBTEST.COM
I20260620 23:10:50.446944 12055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:50.432663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48076 (local address 127.10.191.193:34861)
0620 23:10:50.432925 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:50.432932 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:50.433193 (+ 261us) server_negotiation.cc:407] Connection header received
0620 23:10:50.433431 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:50.433437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:50.433507 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:50.433613 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:50.434977 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:50.435683 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:50.436928 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:50.437358 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:50.437495 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:50.441532 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:50.441537 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:50.441540 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:50.441579 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:50.444137 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:50.444846 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:50.444854 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:50.444857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:50.444983 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:50.445337 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:50.445342 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:50.445346 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:50.445715 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:50.445981 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:50.446263 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 23:10:50.446503 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
W20260620 23:10:50.448812 11686 tablet_service.cc:2908] Rejecting scan request for tablet 811dd0ca1de24596b1d379978c9d73ad: Uninitialized: clean time has not yet been initialized
I20260620 23:10:50.452189 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 11643
I20260620 23:10:50.463790 12055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:50.453113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48084 (local address 127.10.191.193:34861)
0620 23:10:50.453257 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:50.453262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:50.453611 (+ 349us) server_negotiation.cc:407] Connection header received
0620 23:10:50.453726 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:50.453729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:50.453801 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:50.453877 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:50.455003 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:50.455650 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:50.456797 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:50.457187 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:50.457290 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:50.459394 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:50.459398 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:50.459402 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:50.459437 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:50.461645 (+ 2208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:50.462224 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:50.462228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:50.462230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:50.462341 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:50.462640 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:50.462643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:50.462645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:50.462984 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:50.463183 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:50.463411 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 23:10:50.463570 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 23:10:50.464382 11766 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:10:50.552662 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 11643
I20260620 23:10:50.568723 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 11777
Jun 20 23:10:50 dist-test-slave-jkbq krb5kdc[11537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.194@KRBTEST.COM
I20260620 23:10:50.582845 12081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:50.569503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43620 (local address 127.10.191.194:36059)
0620 23:10:50.569974 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:50.569978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:50.569995 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:10:50.570044 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:50.570048 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:50.570102 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:50.570195 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:50.571416 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:50.572120 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:50.573323 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:50.573734 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:50.573825 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:50.577608 (+ 3783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:50.577621 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:50.577654 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:50.577685 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:50.580392 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:50.580978 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:50.580985 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:50.580987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:50.581108 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:50.581485 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:50.581488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:50.581490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:50.581802 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:50.582020 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:50.582313 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 23:10:50.582501 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":277,"threads_started":1}
I20260620 23:10:50.583719 11900 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:10:50.673357 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 11777
I20260620 23:10:50.689934 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 11910
Jun 20 23:10:50 dist-test-slave-jkbq krb5kdc[11537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.195@KRBTEST.COM
I20260620 23:10:50.704021 12051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:50.690668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44438 (local address 127.10.191.195:43711)
0620 23:10:50.690847 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:50.690852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:50.690916 (+ 64us) server_negotiation.cc:407] Connection header received
0620 23:10:50.691053 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:50.691057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:50.691117 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:50.691199 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:50.692379 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:50.693118 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:50.694416 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:50.694810 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:50.694991 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:50.699048 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:50.699062 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:50.699077 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:50.699112 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:50.701610 (+ 2498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:50.702223 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:50.702230 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:50.702232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:50.702350 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:50.702713 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:50.702717 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:50.702719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:50.703146 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:50.703380 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:50.703607 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 23:10:50.703767 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 23:10:50.704838 12033 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:10:50.792973 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 11910
I20260620 23:10:50.808820 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 11550
I20260620 23:10:50.809988 11612 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:10:50.916915 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 11550
2026-06-20T23:10:50Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4180 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:10:51 dist-test-slave-jkbq krb5kdc[12087](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:10:51 dist-test-slave-jkbq krb5kdc[12087](info): set up 2 sockets
Jun 20 23:10:51 dist-test-slave-jkbq krb5kdc[12087](info): commencing operation
krb5kdc: starting...
W20260620 23:10:52.006781 11007 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.002s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Jun 20 23:10:52 dist-test-slave-jkbq krb5kdc[12087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:10:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:10:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.191.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
I20260620 23:10:52.186056 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:42087
--webserver_interface=127.10.191.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:40643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:42087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:10:52.498493 12100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:10:52.498757 12100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:10:52.498934 12100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:10:52.505888 12100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:10:52.505978 12100 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 23:10:52.506065 12100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:10:52.506166 12100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:10:52.506248 12100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:10:52.506358 12100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:10:52.515820 12100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:40643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:42087
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:42087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:10:52.516988 12100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:10:52.518307 12100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:10:52.530337 12108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:52.530596 12105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:52.530368 12106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:10:52.531179 12100 server_base.cc:1034] running on GCE node
I20260620 23:10:52.531908 12100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:10:52.533171 12100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:10:52.534415 12100 hybrid_clock.cc:648] HybridClock initialized: now 1781997052534352 us; error 60 us; skew 500 ppm
I20260620 23:10:52.534766 12100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:10:52 dist-test-slave-jkbq krb5kdc[12087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997052, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:10:52.537783 12100 init.cc:377] Logged in from keytab as kudu/127.10.191.254@KRBTEST.COM (short username kudu)
I20260620 23:10:52.539202 12100 webserver.cc:460] Webserver started at http://127.10.191.254:34451/ using document root <none> and password file <none>
I20260620 23:10:52.539884 12100 fs_manager.cc:359] Metadata directory not provided
I20260620 23:10:52.540011 12100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:10:52.540319 12100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:10:52.543133 12100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "af70cc3d88304030b28e33c149ca3aca"
format_stamp: "Formatted at 2026-06-20 23:10:52 on dist-test-slave-jkbq"
server_key: "78c8e3a181b37d173c551303972327b0"
server_key_iv: "0c30ba8061969de7b7424ef05d7d1472"
server_key_version: "encryptionkey@0"
I20260620 23:10:52.543870 12100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "af70cc3d88304030b28e33c149ca3aca"
format_stamp: "Formatted at 2026-06-20 23:10:52 on dist-test-slave-jkbq"
server_key: "78c8e3a181b37d173c551303972327b0"
server_key_iv: "0c30ba8061969de7b7424ef05d7d1472"
server_key_version: "encryptionkey@0"
I20260620 23:10:52.548641 12100 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 23:10:52.551925 12115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:52.553278 12100 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:10:52.553469 12100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "af70cc3d88304030b28e33c149ca3aca"
format_stamp: "Formatted at 2026-06-20 23:10:52 on dist-test-slave-jkbq"
server_key: "78c8e3a181b37d173c551303972327b0"
server_key_iv: "0c30ba8061969de7b7424ef05d7d1472"
server_key_version: "encryptionkey@0"
I20260620 23:10:52.553711 12100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:10:52.568745 12100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:10:52.572891 12100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:10:52.573146 12100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:10:52.594033 12100 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:42087
I20260620 23:10:52.594053 12176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:42087 every 8 connection(s)
I20260620 23:10:52.595402 12100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:10:52.599110 12177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:52.601430 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 12100
I20260620 23:10:52.601662 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:10:52.602465 11007 external_mini_cluster.cc:1428] Setting key 52e2c98bab99573d167f3929bd090d9a
I20260620 23:10:52.611343 12177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af70cc3d88304030b28e33c149ca3aca: Bootstrap starting.
I20260620 23:10:52.615113 12177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af70cc3d88304030b28e33c149ca3aca: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:52.616457 12177 log.cc:824] T 00000000000000000000000000000000 P af70cc3d88304030b28e33c149ca3aca: Log is configured to *not* fsync() on all Append() calls
Jun 20 23:10:52 dist-test-slave-jkbq krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:10:52.619735 12177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af70cc3d88304030b28e33c149ca3aca: No bootstrap required, opened a new log
I20260620 23:10:52.625660 12180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:52.604518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54268 (local address 127.10.191.254:42087)
0620 23:10:52.605325 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:52.605336 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:52.605376 (+ 40us) server_negotiation.cc:407] Connection header received
0620 23:10:52.607089 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:52.607110 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:52.607486 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:52.607829 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:52.609461 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:52.610737 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:52.612480 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:52.613205 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:52.613354 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:52.618377 (+ 5023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:52.618386 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:52.618401 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:52.618435 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:52.621674 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:52.622477 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:52.622483 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:52.622489 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:52.622657 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:52.623067 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:52.623070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:52.623072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:52.623588 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:52.623800 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:52.624229 (+ 429us) server_negotiation.cc:299] Negotiation successful
0620 23:10:52.624617 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":390,"threads_started":1}
I20260620 23:10:52.628899 12177 raft_consensus.cc:348] T 00000000000000000000000000000000 P af70cc3d88304030b28e33c149ca3aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af70cc3d88304030b28e33c149ca3aca" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 42087 } }
I20260620 23:10:52.629228 12177 raft_consensus.cc:374] T 00000000000000000000000000000000 P af70cc3d88304030b28e33c149ca3aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:10:52.629345 12177 raft_consensus.cc:729] T 00000000000000000000000000000000 P af70cc3d88304030b28e33c149ca3aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af70cc3d88304030b28e33c149ca3aca, State: Initialized, Role: FOLLOWER
I20260620 23:10:52.629894 12177 consensus_queue.cc:260] T 00000000000000000000000000000000 P af70cc3d88304030b28e33c149ca3aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af70cc3d88304030b28e33c149ca3aca" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 42087 } }
I20260620 23:10:52.630129 12177 raft_consensus.cc:388] T 00000000000000000000000000000000 P af70cc3d88304030b28e33c149ca3aca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:10:52.630252 12177 raft_consensus.cc:482] T 00000000000000000000000000000000 P af70cc3d88304030b28e33c149ca3aca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:10:52.630419 12177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af70cc3d88304030b28e33c149ca3aca [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:10:52.632946 12177 raft_consensus.cc:504] T 00000000000000000000000000000000 P af70cc3d88304030b28e33c149ca3aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af70cc3d88304030b28e33c149ca3aca" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 42087 } }
I20260620 23:10:52.633426 12177 leader_election.cc:302] T 00000000000000000000000000000000 P af70cc3d88304030b28e33c149ca3aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af70cc3d88304030b28e33c149ca3aca; no voters:
I20260620 23:10:52.633947 12177 leader_election.cc:288] T 00000000000000000000000000000000 P af70cc3d88304030b28e33c149ca3aca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:10:52.634074 12182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af70cc3d88304030b28e33c149ca3aca [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:10:52.634810 12182 raft_consensus.cc:686] T 00000000000000000000000000000000 P af70cc3d88304030b28e33c149ca3aca [term 1 LEADER]: Becoming Leader. State: Replica: af70cc3d88304030b28e33c149ca3aca, State: Running, Role: LEADER
I20260620 23:10:52.635376 12182 consensus_queue.cc:237] T 00000000000000000000000000000000 P af70cc3d88304030b28e33c149ca3aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af70cc3d88304030b28e33c149ca3aca" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 42087 } }
I20260620 23:10:52.636026 12177 sys_catalog.cc:564] T 00000000000000000000000000000000 P af70cc3d88304030b28e33c149ca3aca [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:10:52.638325 12184 sys_catalog.cc:455] T 00000000000000000000000000000000 P af70cc3d88304030b28e33c149ca3aca [sys.catalog]: SysCatalogTable state changed. Reason: New leader af70cc3d88304030b28e33c149ca3aca. Latest consensus state: current_term: 1 leader_uuid: "af70cc3d88304030b28e33c149ca3aca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af70cc3d88304030b28e33c149ca3aca" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 42087 } } }
I20260620 23:10:52.638742 12184 sys_catalog.cc:458] T 00000000000000000000000000000000 P af70cc3d88304030b28e33c149ca3aca [sys.catalog]: This master's current role is: LEADER
I20260620 23:10:52.638710 12183 sys_catalog.cc:455] T 00000000000000000000000000000000 P af70cc3d88304030b28e33c149ca3aca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af70cc3d88304030b28e33c149ca3aca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af70cc3d88304030b28e33c149ca3aca" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 42087 } } }
I20260620 23:10:52.639091 12183 sys_catalog.cc:458] T 00000000000000000000000000000000 P af70cc3d88304030b28e33c149ca3aca [sys.catalog]: This master's current role is: LEADER
I20260620 23:10:52.640512 12191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:10:52.645731 12191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:10:52.653796 12191 catalog_manager.cc:1347] Generated new cluster ID: 0ab7aa01ea1f4b9ca9949a9ec6b04a9d
I20260620 23:10:52.653959 12191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:10:52.662632 12191 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:10:52.663882 12191 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:10:52.673446 12191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af70cc3d88304030b28e33c149ca3aca: Generated new TSK 0
I20260620 23:10:52.674216 12191 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.191.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab.
Entry for principal kudu/127.10.191.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab.
I20260620 23:10:52.718101 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.193:0
--local_ip_for_outbound_sockets=127.10.191.193
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:42087
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:40643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:10:53.036275 12203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:10:53.036582 12203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:10:53.036770 12203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:10:53.044158 12203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:10:53.044265 12203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:10:53.044430 12203 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:10:53.044555 12203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.193
I20260620 23:10:53.054206 12203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:40643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.191.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.191.254:42087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.193
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:10:53.055547 12203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:10:53.056918 12203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:10:53.070240 12211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:53.070353 12208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:53.070289 12209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:10:53.071197 12203 server_base.cc:1034] running on GCE node
I20260620 23:10:53.071806 12203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:10:53.072666 12203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:10:53.073935 12203 hybrid_clock.cc:648] HybridClock initialized: now 1781997053073877 us; error 60 us; skew 500 ppm
I20260620 23:10:53.074245 12203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:10:53 dist-test-slave-jkbq krb5kdc[12087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997053, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:10:53.077489 12203 init.cc:377] Logged in from keytab as kudu/127.10.191.193@KRBTEST.COM (short username kudu)
I20260620 23:10:53.078856 12203 webserver.cc:460] Webserver started at http://127.10.191.193:33017/ using document root <none> and password file <none>
I20260620 23:10:53.079609 12203 fs_manager.cc:359] Metadata directory not provided
I20260620 23:10:53.079739 12203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:10:53.080051 12203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:10:53.082831 12203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-0/data/instance:
uuid: "ef34c1deb4ab41e1890a653d8d8e659b"
format_stamp: "Formatted at 2026-06-20 23:10:53 on dist-test-slave-jkbq"
server_key: "ca9ee7f37d2fc24926d748cb0e869f19"
server_key_iv: "cae6705b02aff37f27653327e17259b1"
server_key_version: "encryptionkey@0"
I20260620 23:10:53.083806 12203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance:
uuid: "ef34c1deb4ab41e1890a653d8d8e659b"
format_stamp: "Formatted at 2026-06-20 23:10:53 on dist-test-slave-jkbq"
server_key: "ca9ee7f37d2fc24926d748cb0e869f19"
server_key_iv: "cae6705b02aff37f27653327e17259b1"
server_key_version: "encryptionkey@0"
I20260620 23:10:53.088884 12203 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 23:10:53.092360 12218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:53.093752 12203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:10:53.093928 12203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-0/wal
uuid: "ef34c1deb4ab41e1890a653d8d8e659b"
format_stamp: "Formatted at 2026-06-20 23:10:53 on dist-test-slave-jkbq"
server_key: "ca9ee7f37d2fc24926d748cb0e869f19"
server_key_iv: "cae6705b02aff37f27653327e17259b1"
server_key_version: "encryptionkey@0"
I20260620 23:10:53.094141 12203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:10:53.110827 12203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:10:53.115129 12203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:10:53.115406 12203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:10:53.117937 12203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:10:53.118053 12203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:53.118204 12203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:10:53.118278 12203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:10:53 dist-test-slave-jkbq krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997053, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:10:53.143433 12180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:53.118950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:56161 (local address 127.10.191.254:42087)
0620 23:10:53.119097 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:53.119102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:53.120881 (+ 1779us) server_negotiation.cc:407] Connection header received
0620 23:10:53.123128 (+ 2247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:53.123134 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:53.123232 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:53.123359 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:53.126137 (+ 2778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:53.127082 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:53.130268 (+ 3186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:53.130796 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:53.130965 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:53.136024 (+ 5059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:53.136029 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:53.136032 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:53.136065 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:53.140105 (+ 4040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:53.141089 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:53.141097 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:53.141103 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:53.141271 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:53.141716 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:53.141722 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:53.141726 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:53.142008 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:53.142152 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:53.142848 (+ 696us) server_negotiation.cc:299] Negotiation successful
0620 23:10:53.143096 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 23:10:53.144888 12227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:53.119878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:42087 (local address 127.10.191.193:56161)
0620 23:10:53.120715 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:53.120762 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:53.122726 (+ 1964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:53.123725 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:53.123744 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:53.124808 (+ 1064us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:10:53.125850 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:53.125886 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:53.127291 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:53.127299 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:53.130080 (+ 2781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:53.130105 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:53.131100 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:53.131109 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:53.131489 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:53.132460 (+ 971us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:10:53.132502 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:10:53.135781 (+ 3279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:10:53.140246 (+ 4465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:53.140254 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:53.140271 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:53.140961 (+ 690us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:53.141389 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:53.141394 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:53.141397 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:53.141598 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:53.142141 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:10:53.142149 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:10:53.142490 (+ 341us) client_negotiation.cc:769] Sending connection context
0620 23:10:53.142775 (+ 285us) client_negotiation.cc:240] Negotiation successful
0620 23:10:53.143368 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":379,"threads_started":1}
I20260620 23:10:53.165377 12203 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.193:39791
I20260620 23:10:53.165402 12337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.193:39791 every 8 connection(s)
I20260620 23:10:53.166782 12203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
I20260620 23:10:53.169528 12314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:53.153274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:42087 (local address 127.10.191.193:41061)
0620 23:10:53.154096 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:53.154123 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:53.154298 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:53.154686 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:53.154694 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:53.155430 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:10:53.156143 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:53.156162 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:53.157326 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:53.157334 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:53.159536 (+ 2202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:53.159560 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:53.160996 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:53.161008 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:53.161244 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:53.162148 (+ 904us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:10:53.162177 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:10:53.163489 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:10:53.166603 (+ 3114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:53.166608 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:53.166612 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:53.167147 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:53.167604 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:53.167612 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:53.167618 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:53.167836 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:53.168428 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:10:53.168433 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:10:53.168605 (+ 172us) client_negotiation.cc:769] Sending connection context
0620 23:10:53.168775 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 23:10:53.169064 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":448,"threads_started":1}
I20260620 23:10:53.169865 12180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:53.153189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:41061 (local address 127.10.191.254:42087)
0620 23:10:53.153386 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:53.153394 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:53.154275 (+ 881us) server_negotiation.cc:407] Connection header received
0620 23:10:53.154438 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:53.154444 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:53.154520 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:53.154651 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:53.156333 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:53.157182 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:53.159741 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:53.160533 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:53.160657 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:53.163685 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:53.163694 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:53.163697 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:53.163727 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:53.166472 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:53.167294 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:53.167302 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:53.167307 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:53.167469 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:53.167989 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:53.167994 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:53.167997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:53.168305 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:53.168458 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:53.169337 (+ 879us) server_negotiation.cc:299] Negotiation successful
0620 23:10:53.169497 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 23:10:53.169189 12338 heartbeater.cc:347] Connected to a master server at 127.10.191.254:42087
I20260620 23:10:53.170259 12338 heartbeater.cc:464] Registering TS with master...
I20260620 23:10:53.171046 12338 heartbeater.cc:511] Master 127.10.191.254:42087 requested a full tablet report, sending...
I20260620 23:10:53.173220 12132 ts_manager.cc:194] Registered new tserver with Master: ef34c1deb4ab41e1890a653d8d8e659b (127.10.191.193:39791)
I20260620 23:10:53.174619 12132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.193@KRBTEST.COM'} at 127.10.191.193:56161
I20260620 23:10:53.175251 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 12203
I20260620 23:10:53.175488 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance
I20260620 23:10:53.176290 11007 external_mini_cluster.cc:1428] Setting key e0b4cdd95705e8630cfd62e124acb533
WARNING: no policy specified for kudu/127.10.191.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab.
Entry for principal kudu/127.10.191.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab.
I20260620 23:10:53.214660 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.194:0
--local_ip_for_outbound_sockets=127.10.191.194
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:42087
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:40643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:10:53.526579 12342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:10:53.526836 12342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:10:53.527045 12342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:10:53.534559 12342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:10:53.534650 12342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:10:53.534735 12342 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:10:53.534813 12342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.194
I20260620 23:10:53.544390 12342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:40643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.191.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.191.254:42087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.194
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:10:53.545490 12342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:10:53.546694 12342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:10:53.559736 12348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:53.559793 12347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:53.559697 12350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:10:53.560868 12342 server_base.cc:1034] running on GCE node
I20260620 23:10:53.561432 12342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:10:53.562289 12342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:10:53.563525 12342 hybrid_clock.cc:648] HybridClock initialized: now 1781997053563455 us; error 47 us; skew 500 ppm
I20260620 23:10:53.563833 12342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:10:53 dist-test-slave-jkbq krb5kdc[12087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997053, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:10:53.566742 12342 init.cc:377] Logged in from keytab as kudu/127.10.191.194@KRBTEST.COM (short username kudu)
I20260620 23:10:53.568102 12342 webserver.cc:460] Webserver started at http://127.10.191.194:43169/ using document root <none> and password file <none>
I20260620 23:10:53.568755 12342 fs_manager.cc:359] Metadata directory not provided
I20260620 23:10:53.568892 12342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:10:53.569218 12342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:10:53.571954 12342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-1/data/instance:
uuid: "01dc46e565e4425992677fd968c5cc93"
format_stamp: "Formatted at 2026-06-20 23:10:53 on dist-test-slave-jkbq"
server_key: "02e1f199bb4d3901b9f884794e783db9"
server_key_iv: "646982fd1f742212cdcd42b1c0226100"
server_key_version: "encryptionkey@0"
I20260620 23:10:53.572651 12342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance:
uuid: "01dc46e565e4425992677fd968c5cc93"
format_stamp: "Formatted at 2026-06-20 23:10:53 on dist-test-slave-jkbq"
server_key: "02e1f199bb4d3901b9f884794e783db9"
server_key_iv: "646982fd1f742212cdcd42b1c0226100"
server_key_version: "encryptionkey@0"
I20260620 23:10:53.577644 12342 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:10:53.580878 12357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:53.582165 12342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:10:53.582286 12342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-1/wal
uuid: "01dc46e565e4425992677fd968c5cc93"
format_stamp: "Formatted at 2026-06-20 23:10:53 on dist-test-slave-jkbq"
server_key: "02e1f199bb4d3901b9f884794e783db9"
server_key_iv: "646982fd1f742212cdcd42b1c0226100"
server_key_version: "encryptionkey@0"
I20260620 23:10:53.582455 12342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:10:53.596252 12342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:10:53.600581 12342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:10:53.600859 12342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:10:53.603417 12342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:10:53.603541 12342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:53.603679 12342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:10:53.603765 12342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:10:53 dist-test-slave-jkbq krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997053, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:10:53.627861 12180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:53.605492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:36755 (local address 127.10.191.254:42087)
0620 23:10:53.605685 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:53.605695 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:53.606284 (+ 589us) server_negotiation.cc:407] Connection header received
0620 23:10:53.609133 (+ 2849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:53.609139 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:53.609215 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:53.609291 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:53.611638 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:53.612720 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:53.616113 (+ 3393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:53.616630 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:53.616776 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:53.621290 (+ 4514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:53.621295 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:53.621298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:53.621330 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:53.624031 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:53.624981 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:53.624989 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:53.624994 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:53.625140 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:53.625722 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:53.625728 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:53.625731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:53.626030 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:53.626184 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:53.627288 (+ 1104us) server_negotiation.cc:299] Negotiation successful
0620 23:10:53.627517 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 23:10:53.628839 12366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:53.604945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:42087 (local address 127.10.191.194:36755)
0620 23:10:53.606122 (+ 1177us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:53.606168 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:53.608820 (+ 2652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:53.609734 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:53.609745 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:53.610557 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:10:53.611449 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:53.611473 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:53.612936 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:53.612943 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:53.615933 (+ 2990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:53.615955 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:53.617054 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:53.617059 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:53.617312 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:53.618108 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:10:53.618159 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:10:53.621060 (+ 2901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:10:53.624170 (+ 3110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:53.624179 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:53.624194 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:53.624786 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:53.625264 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:53.625268 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:53.625270 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:53.625559 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:53.626277 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:10:53.626295 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:10:53.626799 (+ 504us) client_negotiation.cc:769] Sending connection context
0620 23:10:53.627198 (+ 399us) client_negotiation.cc:240] Negotiation successful
0620 23:10:53.627731 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":909,"thread_start_us":666,"threads_started":1}
I20260620 23:10:53.652371 12180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:53.636610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:44037 (local address 127.10.191.254:42087)
0620 23:10:53.636797 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:53.636804 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:53.637529 (+ 725us) server_negotiation.cc:407] Connection header received
0620 23:10:53.637654 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:53.637662 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:53.637765 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:53.637899 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:53.638785 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:53.640129 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:53.641541 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:53.642413 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:53.642557 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:53.644884 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:53.644892 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:53.644896 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:53.644930 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:53.649199 (+ 4269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:53.650042 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:53.650050 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:53.650055 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:53.650225 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:53.650675 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:53.650682 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:53.650686 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:53.651199 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:53.651360 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:53.651782 (+ 422us) server_negotiation.cc:299] Negotiation successful
0620 23:10:53.651999 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 23:10:53.652463 12444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:53.636751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:42087 (local address 127.10.191.194:44037)
0620 23:10:53.637422 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:53.637443 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:53.637553 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:53.637950 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:53.637953 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:53.638370 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:10:53.638642 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:53.638650 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:53.640331 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:53.640337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:53.641371 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:53.641408 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:53.642617 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:53.642622 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:53.642790 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:53.643524 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:10:53.643553 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:10:53.644691 (+ 1138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:10:53.649359 (+ 4668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:53.649362 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:53.649365 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:53.649898 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:53.650364 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:53.650375 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:53.650380 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:53.650556 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:53.651323 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:10:53.651328 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:10:53.651523 (+ 195us) client_negotiation.cc:769] Sending connection context
0620 23:10:53.651673 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 23:10:53.651984 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":372,"threads_started":1}
I20260620 23:10:53.662317 12342 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.194:45739
I20260620 23:10:53.662341 12476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.194:45739 every 8 connection(s)
I20260620 23:10:53.663594 12342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
I20260620 23:10:53.665683 12477 heartbeater.cc:347] Connected to a master server at 127.10.191.254:42087
I20260620 23:10:53.665932 12477 heartbeater.cc:464] Registering TS with master...
I20260620 23:10:53.666559 12477 heartbeater.cc:511] Master 127.10.191.254:42087 requested a full tablet report, sending...
I20260620 23:10:53.667910 12132 ts_manager.cc:194] Registered new tserver with Master: 01dc46e565e4425992677fd968c5cc93 (127.10.191.194:45739)
I20260620 23:10:53.668668 12132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.194@KRBTEST.COM'} at 127.10.191.194:36755
I20260620 23:10:53.670727 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 12342
I20260620 23:10:53.670948 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance
I20260620 23:10:53.671370 11007 external_mini_cluster.cc:1428] Setting key 28cbdbb39167132b93d2ae5364521793
WARNING: no policy specified for kudu/127.10.191.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab.
Entry for principal kudu/127.10.191.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab.
I20260620 23:10:53.709815 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.195:0
--local_ip_for_outbound_sockets=127.10.191.195
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:42087
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:40643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:10:54.025239 12481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:10:54.025518 12481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:10:54.025645 12481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:10:54.032845 12481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:10:54.032976 12481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:10:54.033130 12481 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:10:54.033255 12481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.195
I20260620 23:10:54.043102 12481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:40643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.191.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.191.254:42087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.195
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:10:54.044346 12481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:10:54.045694 12481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:10:54.058739 12487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:54.058822 12489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:54.058777 12486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:10:54.064366 12481 server_base.cc:1034] running on GCE node
I20260620 23:10:54.065016 12481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:10:54.065989 12481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:10:54.067263 12481 hybrid_clock.cc:648] HybridClock initialized: now 1781997054067194 us; error 41 us; skew 500 ppm
I20260620 23:10:54.067597 12481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:10:54 dist-test-slave-jkbq krb5kdc[12087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997054, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:10:54.070843 12481 init.cc:377] Logged in from keytab as kudu/127.10.191.195@KRBTEST.COM (short username kudu)
I20260620 23:10:54.072440 12481 webserver.cc:460] Webserver started at http://127.10.191.195:42913/ using document root <none> and password file <none>
I20260620 23:10:54.073165 12481 fs_manager.cc:359] Metadata directory not provided
I20260620 23:10:54.073302 12481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:10:54.073629 12481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:10:54.076579 12481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-2/data/instance:
uuid: "6e5ea13b58b14b078a2cb8470cc95774"
format_stamp: "Formatted at 2026-06-20 23:10:54 on dist-test-slave-jkbq"
server_key: "3120bf8f6b149fe0a0af53a4a23da7c7"
server_key_iv: "d8294697cdd06740e9b455dd3cd16f2e"
server_key_version: "encryptionkey@0"
I20260620 23:10:54.077419 12481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance:
uuid: "6e5ea13b58b14b078a2cb8470cc95774"
format_stamp: "Formatted at 2026-06-20 23:10:54 on dist-test-slave-jkbq"
server_key: "3120bf8f6b149fe0a0af53a4a23da7c7"
server_key_iv: "d8294697cdd06740e9b455dd3cd16f2e"
server_key_version: "encryptionkey@0"
I20260620 23:10:54.082669 12481 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 23:10:54.086134 12496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:54.087745 12481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:10:54.087913 12481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-2/wal
uuid: "6e5ea13b58b14b078a2cb8470cc95774"
format_stamp: "Formatted at 2026-06-20 23:10:54 on dist-test-slave-jkbq"
server_key: "3120bf8f6b149fe0a0af53a4a23da7c7"
server_key_iv: "d8294697cdd06740e9b455dd3cd16f2e"
server_key_version: "encryptionkey@0"
I20260620 23:10:54.088126 12481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:10:54.105396 12481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:10:54.109905 12481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:10:54.110188 12481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:10:54.112922 12481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:10:54.113030 12481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:54.113197 12481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:10:54.113339 12481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Jun 20 23:10:54 dist-test-slave-jkbq krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997054, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:10:54.136850 12180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:54.113412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:46235 (local address 127.10.191.254:42087)
0620 23:10:54.113615 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:54.113622 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:54.115072 (+ 1450us) server_negotiation.cc:407] Connection header received
0620 23:10:54.117128 (+ 2056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:54.117136 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:54.117230 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:54.117368 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:54.119622 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.120946 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.123529 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.124276 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.124434 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:54.129845 (+ 5411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:54.129853 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:54.129860 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:54.129906 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:54.133130 (+ 3224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:54.134215 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:54.134219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:54.134221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:54.134346 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:54.134966 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:54.134974 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:54.134980 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:54.135352 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:54.135522 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:54.136267 (+ 745us) server_negotiation.cc:299] Negotiation successful
0620 23:10:54.136496 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 23:10:54.137940 12505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:54.113978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:42087 (local address 127.10.191.195:46235)
0620 23:10:54.114856 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:54.114942 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:54.116832 (+ 1890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:54.117683 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:54.117695 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:54.118510 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:10:54.119380 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:54.119427 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.121146 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.121153 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:54.123403 (+ 2250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:54.123417 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.124608 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.124618 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:54.124904 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:54.125837 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:10:54.125861 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:10:54.129498 (+ 3637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:10:54.133275 (+ 3777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:54.133288 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:54.133313 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:54.134077 (+ 764us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:54.134455 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:54.134463 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:54.134469 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:54.134804 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:54.135502 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:10:54.135513 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:10:54.135879 (+ 366us) client_negotiation.cc:769] Sending connection context
0620 23:10:54.136166 (+ 287us) client_negotiation.cc:240] Negotiation successful
0620 23:10:54.136718 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":381,"threads_started":1}
I20260620 23:10:54.159044 12180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:54.144679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:34665 (local address 127.10.191.254:42087)
0620 23:10:54.144840 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:54.144844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:54.145358 (+ 514us) server_negotiation.cc:407] Connection header received
0620 23:10:54.145573 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:54.145578 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:54.145650 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:54.145745 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:54.147051 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.148348 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.150647 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.151291 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.151431 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:54.153248 (+ 1817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:54.153261 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:54.153265 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:54.153294 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:54.156007 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:54.156998 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:54.157002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:54.157004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:54.157194 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:54.157618 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:54.157621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:54.157623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:54.157868 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:54.158029 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:54.158535 (+ 506us) server_negotiation.cc:299] Negotiation successful
0620 23:10:54.158684 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:10:54.159524 12570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:54.144490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:42087 (local address 127.10.191.195:34665)
0620 23:10:54.145223 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:54.145255 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:54.145439 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:54.145765 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:54.145774 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:54.146382 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:10:54.146846 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:54.146882 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.148503 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.148510 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:54.150440 (+ 1930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:54.150458 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.151509 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.151515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:54.151675 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:54.152362 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:10:54.152378 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:10:54.153082 (+ 704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:10:54.156138 (+ 3056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:54.156147 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:54.156154 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:54.156851 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:54.157304 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:54.157312 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:54.157318 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:54.157507 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:54.158021 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:10:54.158026 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:10:54.158271 (+ 245us) client_negotiation.cc:769] Sending connection context
0620 23:10:54.158446 (+ 175us) client_negotiation.cc:240] Negotiation successful
0620 23:10:54.158815 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":377,"threads_started":1}
I20260620 23:10:54.171684 12481 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.195:39095
I20260620 23:10:54.171705 12615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.195:39095 every 8 connection(s)
I20260620 23:10:54.172791 12481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
I20260620 23:10:54.174713 12616 heartbeater.cc:347] Connected to a master server at 127.10.191.254:42087
I20260620 23:10:54.174961 12616 heartbeater.cc:464] Registering TS with master...
I20260620 23:10:54.175597 12616 heartbeater.cc:511] Master 127.10.191.254:42087 requested a full tablet report, sending...
I20260620 23:10:54.176067 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 12481
I20260620 23:10:54.176257 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance
I20260620 23:10:54.176795 11007 external_mini_cluster.cc:1428] Setting key 1b0a95a5413eb5ca8a85798e88178ded
I20260620 23:10:54.177101 12132 ts_manager.cc:194] Registered new tserver with Master: 6e5ea13b58b14b078a2cb8470cc95774 (127.10.191.195:39095)
I20260620 23:10:54.177662 12338 heartbeater.cc:503] Master 127.10.191.254:42087 was elected leader, sending a full tablet report...
I20260620 23:10:54.178194 12132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.195@KRBTEST.COM'} at 127.10.191.195:46235
I20260620 23:10:54.179860 11007 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 23:10:54 dist-test-slave-jkbq krb5kdc[12087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997054, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 23:10:54 dist-test-slave-jkbq krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997054, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:10:54.223495 12180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:54.210419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49566 (local address 127.10.191.254:42087)
0620 23:10:54.210621 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:54.210625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:54.210994 (+ 369us) server_negotiation.cc:407] Connection header received
0620 23:10:54.211146 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:54.211150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:54.211235 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:54.211333 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:54.212639 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.213382 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.214638 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.215184 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.215292 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:54.219063 (+ 3771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:54.219072 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:54.219075 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:54.219112 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:54.221353 (+ 2241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:54.221908 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:54.221912 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:54.221914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:54.222021 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:54.222332 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:54.222335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:54.222337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:54.222585 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:54.222761 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:54.223119 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 23:10:54.223275 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 23:10:54.227701 12132 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49566:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:10:54.234627 12132 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:10:54.257144 12630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:54.246698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:39791 (local address 127.0.0.1:47716)
0620 23:10:54.247708 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:54.247771 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:54.247969 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:54.249601 (+ 1632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:54.249608 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:54.249643 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:10:54.250100 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:54.250113 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.252036 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.252040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:54.254115 (+ 2075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:54.254130 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.256070 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.256077 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:54.256283 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:54.256344 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 23:10:54.256469 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 23:10:54.256665 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"mutex_wait_us":35,"thread_start_us":733,"threads_started":1}
I20260620 23:10:54.257824 12629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:54.246775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.195:39095 (local address 127.0.0.1:40676)
0620 23:10:54.248716 (+ 1941us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:54.248744 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:54.248863 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:54.251019 (+ 2156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:54.251024 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:54.251050 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:10:54.251389 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:54.251399 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.253481 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.253487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:54.255361 (+ 1874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:54.255373 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.257050 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.257058 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:54.257237 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:54.257273 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 23:10:54.257334 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 23:10:54.257469 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1772,"mutex_wait_us":46,"thread_start_us":1595,"threads_started":1}
I20260620 23:10:54.258301 12631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:54.247231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47716 (local address 127.10.191.193:39791)
0620 23:10:54.248928 (+ 1697us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:54.248949 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:54.248978 (+ 29us) server_negotiation.cc:407] Connection header received
0620 23:10:54.249084 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:54.249095 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:54.249374 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:54.250323 (+ 949us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:10:54.250543 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.251899 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.254280 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.255833 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.257515 (+ 1682us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:54.257692 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:54.257767 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 23:10:54.257934 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1540,"thread_start_us":1721,"threads_started":1}
I20260620 23:10:54.258790 12632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:54.247330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40676 (local address 127.10.191.195:39095)
0620 23:10:54.250461 (+ 3131us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:54.250468 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:54.250500 (+ 32us) server_negotiation.cc:407] Connection header received
0620 23:10:54.250588 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:54.250595 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:54.250813 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:54.251011 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:10:54.251613 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.253339 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.255511 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.256814 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.258140 (+ 1326us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:54.258306 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:54.258403 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 23:10:54.258569 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2959,"thread_start_us":2784,"threads_started":1}
I20260620 23:10:54.259385 12634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:54.248949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43356 (local address 127.10.191.194:45739)
0620 23:10:54.250663 (+ 1714us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:54.250671 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:54.250694 (+ 23us) server_negotiation.cc:407] Connection header received
0620 23:10:54.250794 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:54.250801 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:54.251020 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:54.251205 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:10:54.251918 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.254268 (+ 2350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.256699 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.258030 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.258200 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:54.258395 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:54.258944 (+ 549us) server_negotiation.cc:299] Negotiation successful
0620 23:10:54.259112 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1582,"thread_start_us":1495,"threads_started":1}
I20260620 23:10:54.259418 12633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:54.248400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.194:45739 (local address 127.0.0.1:43356)
0620 23:10:54.249641 (+ 1241us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:54.249667 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:54.249810 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:54.251314 (+ 1504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:54.251318 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:54.251348 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:10:54.251744 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:54.251759 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.254406 (+ 2647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.254412 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:54.256537 (+ 2125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:54.256549 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.258528 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.258537 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:54.258709 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:54.258738 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 23:10:54.258832 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 23:10:54.258966 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1096,"mutex_wait_us":1,"thread_start_us":925,"threads_started":1}
I20260620 23:10:54.262677 12407 tablet_service.cc:1467] Processing CreateTablet for tablet b24c4eeef75642c69b9e9447a270a9a1 (DEFAULT_TABLE table=test-table [id=ba51278593124063b05981a6877df489]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:10:54.262715 12268 tablet_service.cc:1467] Processing CreateTablet for tablet b24c4eeef75642c69b9e9447a270a9a1 (DEFAULT_TABLE table=test-table [id=ba51278593124063b05981a6877df489]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:10:54.262715 12546 tablet_service.cc:1467] Processing CreateTablet for tablet b24c4eeef75642c69b9e9447a270a9a1 (DEFAULT_TABLE table=test-table [id=ba51278593124063b05981a6877df489]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:10:54.264024 12407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b24c4eeef75642c69b9e9447a270a9a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:54.264180 12268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b24c4eeef75642c69b9e9447a270a9a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:54.264185 12546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b24c4eeef75642c69b9e9447a270a9a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:54.275324 12635 tablet_bootstrap.cc:492] T b24c4eeef75642c69b9e9447a270a9a1 P 01dc46e565e4425992677fd968c5cc93: Bootstrap starting.
I20260620 23:10:54.278290 12635 tablet_bootstrap.cc:654] T b24c4eeef75642c69b9e9447a270a9a1 P 01dc46e565e4425992677fd968c5cc93: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:54.279597 12635 log.cc:824] T b24c4eeef75642c69b9e9447a270a9a1 P 01dc46e565e4425992677fd968c5cc93: Log is configured to *not* fsync() on all Append() calls
I20260620 23:10:54.280618 12636 tablet_bootstrap.cc:492] T b24c4eeef75642c69b9e9447a270a9a1 P ef34c1deb4ab41e1890a653d8d8e659b: Bootstrap starting.
I20260620 23:10:54.285156 12636 tablet_bootstrap.cc:654] T b24c4eeef75642c69b9e9447a270a9a1 P ef34c1deb4ab41e1890a653d8d8e659b: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:54.286769 12636 log.cc:824] T b24c4eeef75642c69b9e9447a270a9a1 P ef34c1deb4ab41e1890a653d8d8e659b: Log is configured to *not* fsync() on all Append() calls
I20260620 23:10:54.287158 12637 tablet_bootstrap.cc:492] T b24c4eeef75642c69b9e9447a270a9a1 P 6e5ea13b58b14b078a2cb8470cc95774: Bootstrap starting.
I20260620 23:10:54.292085 12635 tablet_bootstrap.cc:492] T b24c4eeef75642c69b9e9447a270a9a1 P 01dc46e565e4425992677fd968c5cc93: No bootstrap required, opened a new log
I20260620 23:10:54.292388 12635 ts_tablet_manager.cc:1397] T b24c4eeef75642c69b9e9447a270a9a1 P 01dc46e565e4425992677fd968c5cc93: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.004s
I20260620 23:10:54.292781 12637 tablet_bootstrap.cc:654] T b24c4eeef75642c69b9e9447a270a9a1 P 6e5ea13b58b14b078a2cb8470cc95774: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:54.293749 12637 log.cc:824] T b24c4eeef75642c69b9e9447a270a9a1 P 6e5ea13b58b14b078a2cb8470cc95774: Log is configured to *not* fsync() on all Append() calls
I20260620 23:10:54.296460 12637 tablet_bootstrap.cc:492] T b24c4eeef75642c69b9e9447a270a9a1 P 6e5ea13b58b14b078a2cb8470cc95774: No bootstrap required, opened a new log
I20260620 23:10:54.296727 12637 ts_tablet_manager.cc:1397] T b24c4eeef75642c69b9e9447a270a9a1 P 6e5ea13b58b14b078a2cb8470cc95774: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260620 23:10:54.297050 12636 tablet_bootstrap.cc:492] T b24c4eeef75642c69b9e9447a270a9a1 P ef34c1deb4ab41e1890a653d8d8e659b: No bootstrap required, opened a new log
I20260620 23:10:54.297302 12636 ts_tablet_manager.cc:1397] T b24c4eeef75642c69b9e9447a270a9a1 P ef34c1deb4ab41e1890a653d8d8e659b: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.000s
I20260620 23:10:54.302043 12635 raft_consensus.cc:348] T b24c4eeef75642c69b9e9447a270a9a1 P 01dc46e565e4425992677fd968c5cc93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39791 } } peers { permanent_uuid: "6e5ea13b58b14b078a2cb8470cc95774" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39095 } } peers { permanent_uuid: "01dc46e565e4425992677fd968c5cc93" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45739 } }
I20260620 23:10:54.302470 12635 raft_consensus.cc:374] T b24c4eeef75642c69b9e9447a270a9a1 P 01dc46e565e4425992677fd968c5cc93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:10:54.302618 12635 raft_consensus.cc:729] T b24c4eeef75642c69b9e9447a270a9a1 P 01dc46e565e4425992677fd968c5cc93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01dc46e565e4425992677fd968c5cc93, State: Initialized, Role: FOLLOWER
I20260620 23:10:54.303270 12635 consensus_queue.cc:260] T b24c4eeef75642c69b9e9447a270a9a1 P 01dc46e565e4425992677fd968c5cc93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39791 } } peers { permanent_uuid: "6e5ea13b58b14b078a2cb8470cc95774" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39095 } } peers { permanent_uuid: "01dc46e565e4425992677fd968c5cc93" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45739 } }
I20260620 23:10:54.304418 12477 heartbeater.cc:503] Master 127.10.191.254:42087 was elected leader, sending a full tablet report...
I20260620 23:10:54.304967 12635 ts_tablet_manager.cc:1428] T b24c4eeef75642c69b9e9447a270a9a1 P 01dc46e565e4425992677fd968c5cc93: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260620 23:10:54.306830 12637 raft_consensus.cc:348] T b24c4eeef75642c69b9e9447a270a9a1 P 6e5ea13b58b14b078a2cb8470cc95774 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39791 } } peers { permanent_uuid: "6e5ea13b58b14b078a2cb8470cc95774" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39095 } } peers { permanent_uuid: "01dc46e565e4425992677fd968c5cc93" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45739 } }
I20260620 23:10:54.307267 12637 raft_consensus.cc:374] T b24c4eeef75642c69b9e9447a270a9a1 P 6e5ea13b58b14b078a2cb8470cc95774 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:10:54.307382 12637 raft_consensus.cc:729] T b24c4eeef75642c69b9e9447a270a9a1 P 6e5ea13b58b14b078a2cb8470cc95774 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e5ea13b58b14b078a2cb8470cc95774, State: Initialized, Role: FOLLOWER
I20260620 23:10:54.307369 12636 raft_consensus.cc:348] T b24c4eeef75642c69b9e9447a270a9a1 P ef34c1deb4ab41e1890a653d8d8e659b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39791 } } peers { permanent_uuid: "6e5ea13b58b14b078a2cb8470cc95774" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39095 } } peers { permanent_uuid: "01dc46e565e4425992677fd968c5cc93" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45739 } }
I20260620 23:10:54.307811 12636 raft_consensus.cc:374] T b24c4eeef75642c69b9e9447a270a9a1 P ef34c1deb4ab41e1890a653d8d8e659b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:10:54.307933 12636 raft_consensus.cc:729] T b24c4eeef75642c69b9e9447a270a9a1 P ef34c1deb4ab41e1890a653d8d8e659b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef34c1deb4ab41e1890a653d8d8e659b, State: Initialized, Role: FOLLOWER
I20260620 23:10:54.307966 12637 consensus_queue.cc:260] T b24c4eeef75642c69b9e9447a270a9a1 P 6e5ea13b58b14b078a2cb8470cc95774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39791 } } peers { permanent_uuid: "6e5ea13b58b14b078a2cb8470cc95774" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39095 } } peers { permanent_uuid: "01dc46e565e4425992677fd968c5cc93" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45739 } }
I20260620 23:10:54.308530 12636 consensus_queue.cc:260] T b24c4eeef75642c69b9e9447a270a9a1 P ef34c1deb4ab41e1890a653d8d8e659b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39791 } } peers { permanent_uuid: "6e5ea13b58b14b078a2cb8470cc95774" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39095 } } peers { permanent_uuid: "01dc46e565e4425992677fd968c5cc93" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45739 } }
I20260620 23:10:54.310592 12636 ts_tablet_manager.cc:1428] T b24c4eeef75642c69b9e9447a270a9a1 P ef34c1deb4ab41e1890a653d8d8e659b: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260620 23:10:54.311635 12616 heartbeater.cc:503] Master 127.10.191.254:42087 was elected leader, sending a full tablet report...
I20260620 23:10:54.312084 12637 ts_tablet_manager.cc:1428] T b24c4eeef75642c69b9e9447a270a9a1 P 6e5ea13b58b14b078a2cb8470cc95774: Time spent starting tablet: real 0.015s user 0.008s sys 0.008s
I20260620 23:10:54.378324 12641 raft_consensus.cc:482] T b24c4eeef75642c69b9e9447a270a9a1 P 01dc46e565e4425992677fd968c5cc93 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:10:54.378623 12641 raft_consensus.cc:504] T b24c4eeef75642c69b9e9447a270a9a1 P 01dc46e565e4425992677fd968c5cc93 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39791 } } peers { permanent_uuid: "6e5ea13b58b14b078a2cb8470cc95774" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39095 } } peers { permanent_uuid: "01dc46e565e4425992677fd968c5cc93" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45739 } }
I20260620 23:10:54.380277 12641 leader_election.cc:288] T b24c4eeef75642c69b9e9447a270a9a1 P 01dc46e565e4425992677fd968c5cc93 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef34c1deb4ab41e1890a653d8d8e659b (127.10.191.193:39791), 6e5ea13b58b14b078a2cb8470cc95774 (127.10.191.195:39095)
I20260620 23:10:54.389715 12632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:54.380871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:38339 (local address 127.10.191.195:39095)
0620 23:10:54.381063 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:54.381070 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:54.381874 (+ 804us) server_negotiation.cc:407] Connection header received
0620 23:10:54.381995 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:54.382001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:54.382086 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:54.382202 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:10:54.383166 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.384342 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.387448 (+ 3106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.388681 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.388856 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:54.388933 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:54.389339 (+ 406us) server_negotiation.cc:299] Negotiation successful
0620 23:10:54.389466 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 23:10:54.389716 12631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:54.380868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:59309 (local address 127.10.191.193:39791)
0620 23:10:54.381064 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:54.381070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:54.381805 (+ 735us) server_negotiation.cc:407] Connection header received
0620 23:10:54.382022 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:54.382027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:54.382104 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:54.382202 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:10:54.383097 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.384348 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.387456 (+ 3108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.388681 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.388881 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:54.388950 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:54.389335 (+ 385us) server_negotiation.cc:299] Negotiation successful
0620 23:10:54.389466 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 23:10:54.390009 12644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:54.380653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:39791 (local address 127.10.191.194:59309)
0620 23:10:54.381648 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:54.381672 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:54.381843 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:54.382227 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:54.382232 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:54.382264 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:10:54.382849 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:54.382898 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.384528 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.384534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:54.387257 (+ 2723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:54.387270 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.388942 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.388949 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:54.389141 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:54.389196 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 23:10:54.389293 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 23:10:54.389486 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":849,"mutex_wait_us":349,"spinlock_wait_cycles":17792,"thread_start_us":617,"threads_started":1}
I20260620 23:10:54.390033 12645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:54.380748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.195:39095 (local address 127.10.191.194:38339)
0620 23:10:54.381676 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:54.381690 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:54.381843 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:54.382227 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:54.382232 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:54.382267 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:10:54.382980 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:54.382993 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.384529 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.384534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:54.387256 (+ 2722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:54.387270 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.388952 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.388957 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:54.389148 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:54.389200 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 23:10:54.389293 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 23:10:54.389486 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"mutex_wait_us":230,"spinlock_wait_cycles":22400,"thread_start_us":530,"threads_started":1}
I20260620 23:10:54.391234 12571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b24c4eeef75642c69b9e9447a270a9a1" candidate_uuid: "01dc46e565e4425992677fd968c5cc93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e5ea13b58b14b078a2cb8470cc95774" is_pre_election: true
I20260620 23:10:54.391263 12288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b24c4eeef75642c69b9e9447a270a9a1" candidate_uuid: "01dc46e565e4425992677fd968c5cc93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" is_pre_election: true
I20260620 23:10:54.391728 12571 raft_consensus.cc:2454] T b24c4eeef75642c69b9e9447a270a9a1 P 6e5ea13b58b14b078a2cb8470cc95774 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01dc46e565e4425992677fd968c5cc93 in term 0.
I20260620 23:10:54.391757 12288 raft_consensus.cc:2454] T b24c4eeef75642c69b9e9447a270a9a1 P ef34c1deb4ab41e1890a653d8d8e659b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01dc46e565e4425992677fd968c5cc93 in term 0.
I20260620 23:10:54.392416 12361 leader_election.cc:302] T b24c4eeef75642c69b9e9447a270a9a1 P 01dc46e565e4425992677fd968c5cc93 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 01dc46e565e4425992677fd968c5cc93, 6e5ea13b58b14b078a2cb8470cc95774, ef34c1deb4ab41e1890a653d8d8e659b; no voters:
I20260620 23:10:54.392838 12641 raft_consensus.cc:2781] T b24c4eeef75642c69b9e9447a270a9a1 P 01dc46e565e4425992677fd968c5cc93 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:10:54.392978 12641 raft_consensus.cc:482] T b24c4eeef75642c69b9e9447a270a9a1 P 01dc46e565e4425992677fd968c5cc93 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:10:54.393100 12641 raft_consensus.cc:3037] T b24c4eeef75642c69b9e9447a270a9a1 P 01dc46e565e4425992677fd968c5cc93 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:10:54.395365 12641 raft_consensus.cc:504] T b24c4eeef75642c69b9e9447a270a9a1 P 01dc46e565e4425992677fd968c5cc93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39791 } } peers { permanent_uuid: "6e5ea13b58b14b078a2cb8470cc95774" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39095 } } peers { permanent_uuid: "01dc46e565e4425992677fd968c5cc93" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45739 } }
I20260620 23:10:54.395977 12641 leader_election.cc:288] T b24c4eeef75642c69b9e9447a270a9a1 P 01dc46e565e4425992677fd968c5cc93 [CANDIDATE]: Term 1 election: Requested vote from peers ef34c1deb4ab41e1890a653d8d8e659b (127.10.191.193:39791), 6e5ea13b58b14b078a2cb8470cc95774 (127.10.191.195:39095)
I20260620 23:10:54.396451 12571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b24c4eeef75642c69b9e9447a270a9a1" candidate_uuid: "01dc46e565e4425992677fd968c5cc93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e5ea13b58b14b078a2cb8470cc95774"
I20260620 23:10:54.396459 12288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b24c4eeef75642c69b9e9447a270a9a1" candidate_uuid: "01dc46e565e4425992677fd968c5cc93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef34c1deb4ab41e1890a653d8d8e659b"
I20260620 23:10:54.396647 12571 raft_consensus.cc:3037] T b24c4eeef75642c69b9e9447a270a9a1 P 6e5ea13b58b14b078a2cb8470cc95774 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:10:54.396664 12288 raft_consensus.cc:3037] T b24c4eeef75642c69b9e9447a270a9a1 P ef34c1deb4ab41e1890a653d8d8e659b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:10:54.399080 12288 raft_consensus.cc:2454] T b24c4eeef75642c69b9e9447a270a9a1 P ef34c1deb4ab41e1890a653d8d8e659b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01dc46e565e4425992677fd968c5cc93 in term 1.
I20260620 23:10:54.399091 12571 raft_consensus.cc:2454] T b24c4eeef75642c69b9e9447a270a9a1 P 6e5ea13b58b14b078a2cb8470cc95774 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01dc46e565e4425992677fd968c5cc93 in term 1.
I20260620 23:10:54.399468 12361 leader_election.cc:302] T b24c4eeef75642c69b9e9447a270a9a1 P 01dc46e565e4425992677fd968c5cc93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01dc46e565e4425992677fd968c5cc93, ef34c1deb4ab41e1890a653d8d8e659b; no voters:
I20260620 23:10:54.399797 12641 raft_consensus.cc:2781] T b24c4eeef75642c69b9e9447a270a9a1 P 01dc46e565e4425992677fd968c5cc93 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:10:54.400228 12641 raft_consensus.cc:686] T b24c4eeef75642c69b9e9447a270a9a1 P 01dc46e565e4425992677fd968c5cc93 [term 1 LEADER]: Becoming Leader. State: Replica: 01dc46e565e4425992677fd968c5cc93, State: Running, Role: LEADER
I20260620 23:10:54.400779 12641 consensus_queue.cc:237] T b24c4eeef75642c69b9e9447a270a9a1 P 01dc46e565e4425992677fd968c5cc93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39791 } } peers { permanent_uuid: "6e5ea13b58b14b078a2cb8470cc95774" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39095 } } peers { permanent_uuid: "01dc46e565e4425992677fd968c5cc93" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45739 } }
I20260620 23:10:54.405153 12132 catalog_manager.cc:5495] T b24c4eeef75642c69b9e9447a270a9a1 P 01dc46e565e4425992677fd968c5cc93 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01dc46e565e4425992677fd968c5cc93 (127.10.191.194). New cstate: current_term: 1 leader_uuid: "01dc46e565e4425992677fd968c5cc93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e5ea13b58b14b078a2cb8470cc95774" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01dc46e565e4425992677fd968c5cc93" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45739 } health_report { overall_health: HEALTHY } } }
W20260620 23:10:54.415489 12478 tablet.cc:2369] T b24c4eeef75642c69b9e9447a270a9a1 P 01dc46e565e4425992677fd968c5cc93: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:10:54.420311 12339 tablet.cc:2369] T b24c4eeef75642c69b9e9447a270a9a1 P ef34c1deb4ab41e1890a653d8d8e659b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:10:54.424276 12617 tablet.cc:2369] T b24c4eeef75642c69b9e9447a270a9a1 P 6e5ea13b58b14b078a2cb8470cc95774: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:10:54.457949 12634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:54.452334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43364 (local address 127.10.191.194:45739)
0620 23:10:54.452549 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:54.452554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:54.452575 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:10:54.452810 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:54.452814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:54.452891 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:54.453048 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:10:54.453445 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.454181 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.455607 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.456015 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.456151 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:54.456653 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:10:54.457262 (+ 609us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:10:54.457462 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:54.457571 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 23:10:54.457669 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 23:10:54.466162 12571 raft_consensus.cc:1261] T b24c4eeef75642c69b9e9447a270a9a1 P 6e5ea13b58b14b078a2cb8470cc95774 [term 1 FOLLOWER]: Refusing update from remote peer 01dc46e565e4425992677fd968c5cc93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:10:54.466162 12288 raft_consensus.cc:1261] T b24c4eeef75642c69b9e9447a270a9a1 P ef34c1deb4ab41e1890a653d8d8e659b [term 1 FOLLOWER]: Refusing update from remote peer 01dc46e565e4425992677fd968c5cc93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:10:54.467242 12646 consensus_queue.cc:1035] T b24c4eeef75642c69b9e9447a270a9a1 P 01dc46e565e4425992677fd968c5cc93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e5ea13b58b14b078a2cb8470cc95774" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:10:54.467649 12641 consensus_queue.cc:1035] T b24c4eeef75642c69b9e9447a270a9a1 P 01dc46e565e4425992677fd968c5cc93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:10:54.477908 12650 mvcc.cc:205] Tried to move back new op lower bound from 7299059935082622976 to 7299059934829543424. Current Snapshot: MvccSnapshot[applied={T|T < 7299059935082622976}]
I20260620 23:10:54.493606 12132 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:10:54.493839 12132 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:10:54.497306 12132 catalog_manager.cc:5782] T 00000000000000000000000000000000 P af70cc3d88304030b28e33c149ca3aca: Sending DeleteTablet for 3 replicas of tablet b24c4eeef75642c69b9e9447a270a9a1
I20260620 23:10:54.498644 12407 tablet_service.cc:1514] Processing DeleteTablet for tablet b24c4eeef75642c69b9e9447a270a9a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:10:54 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:43356
I20260620 23:10:54.498634 12546 tablet_service.cc:1514] Processing DeleteTablet for tablet b24c4eeef75642c69b9e9447a270a9a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:10:54 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:40676
I20260620 23:10:54.498631 12268 tablet_service.cc:1514] Processing DeleteTablet for tablet b24c4eeef75642c69b9e9447a270a9a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:10:54 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:47716
I20260620 23:10:54.504387 12132 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49566:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:10:54.506006 12132 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:10:54.513518 12661 tablet_replica.cc:330] T b24c4eeef75642c69b9e9447a270a9a1 P 01dc46e565e4425992677fd968c5cc93: stopping tablet replica
I20260620 23:10:54.514338 12661 raft_consensus.cc:2229] T b24c4eeef75642c69b9e9447a270a9a1 P 01dc46e565e4425992677fd968c5cc93 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:10:54.515954 12545 tablet_service.cc:1467] Processing CreateTablet for tablet a052bbcd931449279f88723c2bf6f723 (DEFAULT_TABLE table=test-table [id=53a312f4fa934432bd1416ff50b4738d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:10:54.515990 12267 tablet_service.cc:1467] Processing CreateTablet for tablet a052bbcd931449279f88723c2bf6f723 (DEFAULT_TABLE table=test-table [id=53a312f4fa934432bd1416ff50b4738d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:10:54.516547 12545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a052bbcd931449279f88723c2bf6f723. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:54.516698 12267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a052bbcd931449279f88723c2bf6f723. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:54.523684 12407 tablet_service.cc:1467] Processing CreateTablet for tablet a052bbcd931449279f88723c2bf6f723 (DEFAULT_TABLE table=test-table [id=53a312f4fa934432bd1416ff50b4738d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:10:54.524294 12407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a052bbcd931449279f88723c2bf6f723. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:54.524451 12637 tablet_bootstrap.cc:492] T a052bbcd931449279f88723c2bf6f723 P 6e5ea13b58b14b078a2cb8470cc95774: Bootstrap starting.
I20260620 23:10:54.526130 12637 tablet_bootstrap.cc:654] T a052bbcd931449279f88723c2bf6f723 P 6e5ea13b58b14b078a2cb8470cc95774: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:54.529354 12659 tablet_replica.cc:330] T b24c4eeef75642c69b9e9447a270a9a1 P 6e5ea13b58b14b078a2cb8470cc95774: stopping tablet replica
I20260620 23:10:54.529853 12659 raft_consensus.cc:2229] T b24c4eeef75642c69b9e9447a270a9a1 P 6e5ea13b58b14b078a2cb8470cc95774 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:10:54.530426 12659 raft_consensus.cc:2258] T b24c4eeef75642c69b9e9447a270a9a1 P 6e5ea13b58b14b078a2cb8470cc95774 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:10:54.532701 12659 ts_tablet_manager.cc:1905] T b24c4eeef75642c69b9e9447a270a9a1 P 6e5ea13b58b14b078a2cb8470cc95774: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:10:54.533802 12661 raft_consensus.cc:2258] T b24c4eeef75642c69b9e9447a270a9a1 P 01dc46e565e4425992677fd968c5cc93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:10:54.534457 12636 tablet_bootstrap.cc:492] T a052bbcd931449279f88723c2bf6f723 P ef34c1deb4ab41e1890a653d8d8e659b: Bootstrap starting.
I20260620 23:10:54.536546 12636 tablet_bootstrap.cc:654] T a052bbcd931449279f88723c2bf6f723 P ef34c1deb4ab41e1890a653d8d8e659b: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:54.537544 12661 ts_tablet_manager.cc:1905] T b24c4eeef75642c69b9e9447a270a9a1 P 01dc46e565e4425992677fd968c5cc93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:10:54.541464 12660 tablet_replica.cc:330] T b24c4eeef75642c69b9e9447a270a9a1 P ef34c1deb4ab41e1890a653d8d8e659b: stopping tablet replica
I20260620 23:10:54.543661 12659 ts_tablet_manager.cc:1918] T b24c4eeef75642c69b9e9447a270a9a1 P 6e5ea13b58b14b078a2cb8470cc95774: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:10:54.543710 12660 raft_consensus.cc:2229] T b24c4eeef75642c69b9e9447a270a9a1 P ef34c1deb4ab41e1890a653d8d8e659b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:10:54.544319 12660 raft_consensus.cc:2258] T b24c4eeef75642c69b9e9447a270a9a1 P ef34c1deb4ab41e1890a653d8d8e659b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:10:54.546885 12636 tablet_bootstrap.cc:492] T a052bbcd931449279f88723c2bf6f723 P ef34c1deb4ab41e1890a653d8d8e659b: No bootstrap required, opened a new log
I20260620 23:10:54.547030 12636 ts_tablet_manager.cc:1397] T a052bbcd931449279f88723c2bf6f723 P ef34c1deb4ab41e1890a653d8d8e659b: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.004s
I20260620 23:10:54.547256 12660 ts_tablet_manager.cc:1905] T b24c4eeef75642c69b9e9447a270a9a1 P ef34c1deb4ab41e1890a653d8d8e659b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:10:54.547812 12636 raft_consensus.cc:348] T a052bbcd931449279f88723c2bf6f723 P ef34c1deb4ab41e1890a653d8d8e659b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39791 } } peers { permanent_uuid: "01dc46e565e4425992677fd968c5cc93" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45739 } } peers { permanent_uuid: "6e5ea13b58b14b078a2cb8470cc95774" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39095 } }
I20260620 23:10:54.548048 12636 raft_consensus.cc:374] T a052bbcd931449279f88723c2bf6f723 P ef34c1deb4ab41e1890a653d8d8e659b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:10:54.548135 12636 raft_consensus.cc:729] T a052bbcd931449279f88723c2bf6f723 P ef34c1deb4ab41e1890a653d8d8e659b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef34c1deb4ab41e1890a653d8d8e659b, State: Initialized, Role: FOLLOWER
I20260620 23:10:54.548377 12636 consensus_queue.cc:260] T a052bbcd931449279f88723c2bf6f723 P ef34c1deb4ab41e1890a653d8d8e659b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39791 } } peers { permanent_uuid: "01dc46e565e4425992677fd968c5cc93" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45739 } } peers { permanent_uuid: "6e5ea13b58b14b078a2cb8470cc95774" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39095 } }
I20260620 23:10:54.549043 12636 ts_tablet_manager.cc:1428] T a052bbcd931449279f88723c2bf6f723 P ef34c1deb4ab41e1890a653d8d8e659b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:10:54.550408 12637 tablet_bootstrap.cc:492] T a052bbcd931449279f88723c2bf6f723 P 6e5ea13b58b14b078a2cb8470cc95774: No bootstrap required, opened a new log
I20260620 23:10:54.550516 12659 log.cc:1196] T b24c4eeef75642c69b9e9447a270a9a1 P 6e5ea13b58b14b078a2cb8470cc95774: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-2/wal/wals/b24c4eeef75642c69b9e9447a270a9a1
I20260620 23:10:54.550557 12637 ts_tablet_manager.cc:1397] T a052bbcd931449279f88723c2bf6f723 P 6e5ea13b58b14b078a2cb8470cc95774: Time spent bootstrapping tablet: real 0.026s user 0.017s sys 0.003s
I20260620 23:10:54.551095 12659 ts_tablet_manager.cc:1939] T b24c4eeef75642c69b9e9447a270a9a1 P 6e5ea13b58b14b078a2cb8470cc95774: Deleting consensus metadata
I20260620 23:10:54.551427 12637 raft_consensus.cc:348] T a052bbcd931449279f88723c2bf6f723 P 6e5ea13b58b14b078a2cb8470cc95774 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39791 } } peers { permanent_uuid: "01dc46e565e4425992677fd968c5cc93" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45739 } } peers { permanent_uuid: "6e5ea13b58b14b078a2cb8470cc95774" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39095 } }
I20260620 23:10:54.551640 12637 raft_consensus.cc:374] T a052bbcd931449279f88723c2bf6f723 P 6e5ea13b58b14b078a2cb8470cc95774 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:10:54.551748 12637 raft_consensus.cc:729] T a052bbcd931449279f88723c2bf6f723 P 6e5ea13b58b14b078a2cb8470cc95774 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e5ea13b58b14b078a2cb8470cc95774, State: Initialized, Role: FOLLOWER
I20260620 23:10:54.552064 12637 consensus_queue.cc:260] T a052bbcd931449279f88723c2bf6f723 P 6e5ea13b58b14b078a2cb8470cc95774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39791 } } peers { permanent_uuid: "01dc46e565e4425992677fd968c5cc93" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45739 } } peers { permanent_uuid: "6e5ea13b58b14b078a2cb8470cc95774" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39095 } }
I20260620 23:10:54.553133 12637 ts_tablet_manager.cc:1428] T a052bbcd931449279f88723c2bf6f723 P 6e5ea13b58b14b078a2cb8470cc95774: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:10:54.553890 12635 tablet_bootstrap.cc:492] T a052bbcd931449279f88723c2bf6f723 P 01dc46e565e4425992677fd968c5cc93: Bootstrap starting.
I20260620 23:10:54.555788 12635 tablet_bootstrap.cc:654] T a052bbcd931449279f88723c2bf6f723 P 01dc46e565e4425992677fd968c5cc93: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:54.556391 12661 ts_tablet_manager.cc:1918] T b24c4eeef75642c69b9e9447a270a9a1 P 01dc46e565e4425992677fd968c5cc93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:10:54.556596 12661 log.cc:1196] T b24c4eeef75642c69b9e9447a270a9a1 P 01dc46e565e4425992677fd968c5cc93: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-1/wal/wals/b24c4eeef75642c69b9e9447a270a9a1
I20260620 23:10:54.557011 12116 catalog_manager.cc:4841] TS 6e5ea13b58b14b078a2cb8470cc95774 (127.10.191.195:39095): tablet b24c4eeef75642c69b9e9447a270a9a1 (table test-table [id=ba51278593124063b05981a6877df489]) successfully deleted
I20260620 23:10:54.557217 12661 ts_tablet_manager.cc:1939] T b24c4eeef75642c69b9e9447a270a9a1 P 01dc46e565e4425992677fd968c5cc93: Deleting consensus metadata
I20260620 23:10:54.559278 12119 catalog_manager.cc:4841] TS 01dc46e565e4425992677fd968c5cc93 (127.10.191.194:45739): tablet b24c4eeef75642c69b9e9447a270a9a1 (table test-table [id=ba51278593124063b05981a6877df489]) successfully deleted
I20260620 23:10:54.563305 12660 ts_tablet_manager.cc:1918] T b24c4eeef75642c69b9e9447a270a9a1 P ef34c1deb4ab41e1890a653d8d8e659b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:10:54.563478 12660 log.cc:1196] T b24c4eeef75642c69b9e9447a270a9a1 P ef34c1deb4ab41e1890a653d8d8e659b: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-0/wal/wals/b24c4eeef75642c69b9e9447a270a9a1
I20260620 23:10:54.564025 12660 ts_tablet_manager.cc:1939] T b24c4eeef75642c69b9e9447a270a9a1 P ef34c1deb4ab41e1890a653d8d8e659b: Deleting consensus metadata
I20260620 23:10:54.564285 12635 tablet_bootstrap.cc:492] T a052bbcd931449279f88723c2bf6f723 P 01dc46e565e4425992677fd968c5cc93: No bootstrap required, opened a new log
I20260620 23:10:54.564455 12635 ts_tablet_manager.cc:1397] T a052bbcd931449279f88723c2bf6f723 P 01dc46e565e4425992677fd968c5cc93: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260620 23:10:54.565143 12635 raft_consensus.cc:348] T a052bbcd931449279f88723c2bf6f723 P 01dc46e565e4425992677fd968c5cc93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39791 } } peers { permanent_uuid: "01dc46e565e4425992677fd968c5cc93" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45739 } } peers { permanent_uuid: "6e5ea13b58b14b078a2cb8470cc95774" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39095 } }
I20260620 23:10:54.565347 12635 raft_consensus.cc:374] T a052bbcd931449279f88723c2bf6f723 P 01dc46e565e4425992677fd968c5cc93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:10:54.565455 12635 raft_consensus.cc:729] T a052bbcd931449279f88723c2bf6f723 P 01dc46e565e4425992677fd968c5cc93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01dc46e565e4425992677fd968c5cc93, State: Initialized, Role: FOLLOWER
I20260620 23:10:54.565762 12119 catalog_manager.cc:4841] TS ef34c1deb4ab41e1890a653d8d8e659b (127.10.191.193:39791): tablet b24c4eeef75642c69b9e9447a270a9a1 (table test-table [id=ba51278593124063b05981a6877df489]) successfully deleted
I20260620 23:10:54.565740 12635 consensus_queue.cc:260] T a052bbcd931449279f88723c2bf6f723 P 01dc46e565e4425992677fd968c5cc93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39791 } } peers { permanent_uuid: "01dc46e565e4425992677fd968c5cc93" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45739 } } peers { permanent_uuid: "6e5ea13b58b14b078a2cb8470cc95774" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39095 } }
I20260620 23:10:54.566390 12635 ts_tablet_manager.cc:1428] T a052bbcd931449279f88723c2bf6f723 P 01dc46e565e4425992677fd968c5cc93: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:10:54.566849 12656 raft_consensus.cc:482] T a052bbcd931449279f88723c2bf6f723 P 01dc46e565e4425992677fd968c5cc93 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:10:54.567075 12656 raft_consensus.cc:504] T a052bbcd931449279f88723c2bf6f723 P 01dc46e565e4425992677fd968c5cc93 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39791 } } peers { permanent_uuid: "01dc46e565e4425992677fd968c5cc93" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45739 } } peers { permanent_uuid: "6e5ea13b58b14b078a2cb8470cc95774" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39095 } }
I20260620 23:10:54.567898 12656 leader_election.cc:288] T a052bbcd931449279f88723c2bf6f723 P 01dc46e565e4425992677fd968c5cc93 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef34c1deb4ab41e1890a653d8d8e659b (127.10.191.193:39791), 6e5ea13b58b14b078a2cb8470cc95774 (127.10.191.195:39095)
I20260620 23:10:54.568457 12288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a052bbcd931449279f88723c2bf6f723" candidate_uuid: "01dc46e565e4425992677fd968c5cc93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" is_pre_election: true
I20260620 23:10:54.568550 12571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a052bbcd931449279f88723c2bf6f723" candidate_uuid: "01dc46e565e4425992677fd968c5cc93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e5ea13b58b14b078a2cb8470cc95774" is_pre_election: true
I20260620 23:10:54.568693 12288 raft_consensus.cc:2454] T a052bbcd931449279f88723c2bf6f723 P ef34c1deb4ab41e1890a653d8d8e659b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01dc46e565e4425992677fd968c5cc93 in term 0.
I20260620 23:10:54.568760 12571 raft_consensus.cc:2454] T a052bbcd931449279f88723c2bf6f723 P 6e5ea13b58b14b078a2cb8470cc95774 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01dc46e565e4425992677fd968c5cc93 in term 0.
I20260620 23:10:54.569191 12358 leader_election.cc:302] T a052bbcd931449279f88723c2bf6f723 P 01dc46e565e4425992677fd968c5cc93 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01dc46e565e4425992677fd968c5cc93, 6e5ea13b58b14b078a2cb8470cc95774; no voters:
I20260620 23:10:54.569600 12656 raft_consensus.cc:2781] T a052bbcd931449279f88723c2bf6f723 P 01dc46e565e4425992677fd968c5cc93 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:10:54.569728 12656 raft_consensus.cc:482] T a052bbcd931449279f88723c2bf6f723 P 01dc46e565e4425992677fd968c5cc93 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:10:54.569814 12656 raft_consensus.cc:3037] T a052bbcd931449279f88723c2bf6f723 P 01dc46e565e4425992677fd968c5cc93 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:10:54.572824 12656 raft_consensus.cc:504] T a052bbcd931449279f88723c2bf6f723 P 01dc46e565e4425992677fd968c5cc93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39791 } } peers { permanent_uuid: "01dc46e565e4425992677fd968c5cc93" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45739 } } peers { permanent_uuid: "6e5ea13b58b14b078a2cb8470cc95774" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39095 } }
I20260620 23:10:54.573428 12656 leader_election.cc:288] T a052bbcd931449279f88723c2bf6f723 P 01dc46e565e4425992677fd968c5cc93 [CANDIDATE]: Term 1 election: Requested vote from peers ef34c1deb4ab41e1890a653d8d8e659b (127.10.191.193:39791), 6e5ea13b58b14b078a2cb8470cc95774 (127.10.191.195:39095)
I20260620 23:10:54.573805 12288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a052bbcd931449279f88723c2bf6f723" candidate_uuid: "01dc46e565e4425992677fd968c5cc93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef34c1deb4ab41e1890a653d8d8e659b"
I20260620 23:10:54.574015 12288 raft_consensus.cc:3037] T a052bbcd931449279f88723c2bf6f723 P ef34c1deb4ab41e1890a653d8d8e659b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:10:54.574141 12571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a052bbcd931449279f88723c2bf6f723" candidate_uuid: "01dc46e565e4425992677fd968c5cc93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e5ea13b58b14b078a2cb8470cc95774"
I20260620 23:10:54.574333 12571 raft_consensus.cc:3037] T a052bbcd931449279f88723c2bf6f723 P 6e5ea13b58b14b078a2cb8470cc95774 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:10:54.576056 12288 raft_consensus.cc:2454] T a052bbcd931449279f88723c2bf6f723 P ef34c1deb4ab41e1890a653d8d8e659b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01dc46e565e4425992677fd968c5cc93 in term 1.
I20260620 23:10:54.576465 12361 leader_election.cc:302] T a052bbcd931449279f88723c2bf6f723 P 01dc46e565e4425992677fd968c5cc93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01dc46e565e4425992677fd968c5cc93, ef34c1deb4ab41e1890a653d8d8e659b; no voters:
I20260620 23:10:54.576727 12656 raft_consensus.cc:2781] T a052bbcd931449279f88723c2bf6f723 P 01dc46e565e4425992677fd968c5cc93 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:10:54.576908 12656 raft_consensus.cc:686] T a052bbcd931449279f88723c2bf6f723 P 01dc46e565e4425992677fd968c5cc93 [term 1 LEADER]: Becoming Leader. State: Replica: 01dc46e565e4425992677fd968c5cc93, State: Running, Role: LEADER
I20260620 23:10:54.577131 12571 raft_consensus.cc:2454] T a052bbcd931449279f88723c2bf6f723 P 6e5ea13b58b14b078a2cb8470cc95774 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01dc46e565e4425992677fd968c5cc93 in term 1.
I20260620 23:10:54.577186 12656 consensus_queue.cc:237] T a052bbcd931449279f88723c2bf6f723 P 01dc46e565e4425992677fd968c5cc93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39791 } } peers { permanent_uuid: "01dc46e565e4425992677fd968c5cc93" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45739 } } peers { permanent_uuid: "6e5ea13b58b14b078a2cb8470cc95774" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39095 } }
I20260620 23:10:54.579612 12131 catalog_manager.cc:5495] T a052bbcd931449279f88723c2bf6f723 P 01dc46e565e4425992677fd968c5cc93 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01dc46e565e4425992677fd968c5cc93 (127.10.191.194). New cstate: current_term: 1 leader_uuid: "01dc46e565e4425992677fd968c5cc93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01dc46e565e4425992677fd968c5cc93" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45739 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e5ea13b58b14b078a2cb8470cc95774" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39095 } health_report { overall_health: UNKNOWN } } }
Jun 20 23:10:54 dist-test-slave-jkbq krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997052, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:10:54.623574 12671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:54.609487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:42087 (local address 127.0.0.1:49570)
0620 23:10:54.610210 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:54.610227 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:54.610397 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:54.610721 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:54.610727 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:54.611488 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:10:54.611785 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:54.611795 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.612857 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.612865 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:54.613869 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:54.613880 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.614606 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.614612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:54.614792 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:54.615531 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:10:54.615549 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:10:54.617732 (+ 2183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:10:54.621009 (+ 3277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:54.621022 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:54.621030 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:54.621473 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:54.621817 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:54.621823 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:54.621827 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:54.621958 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:54.622462 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:10:54.622470 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:10:54.622639 (+ 169us) client_negotiation.cc:769] Sending connection context
0620 23:10:54.622808 (+ 169us) client_negotiation.cc:240] Negotiation successful
0620 23:10:54.623112 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":403,"threads_started":1}
I20260620 23:10:54.623581 12180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:54.609488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49570 (local address 127.10.191.254:42087)
0620 23:10:54.609684 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:54.609703 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:54.610314 (+ 611us) server_negotiation.cc:407] Connection header received
0620 23:10:54.610526 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:54.610535 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:54.610588 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:54.610686 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:54.611924 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.612728 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.614007 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.614421 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.614559 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:54.617929 (+ 3370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:54.617934 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:54.617937 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:54.617964 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:54.620886 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:54.621586 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:54.621593 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:54.621598 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:54.621718 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:54.622077 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:54.622081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:54.622085 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:54.622339 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:54.622486 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:54.622928 (+ 442us) server_negotiation.cc:299] Negotiation successful
0620 23:10:54.623234 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":120,"server-negotiator.queue_time_us":67}
I20260620 23:10:54.628479 12131 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:49570:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 23:10:54.629578 12131 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:10:54.636513 12267 tablet_service.cc:1467] Processing CreateTablet for tablet c1bcdbe5695a4fd2b4e514cacb6fcbf4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=995e2a7929d545f8b60371f9db40f0d4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:10:54.636504 12407 tablet_service.cc:1467] Processing CreateTablet for tablet c1bcdbe5695a4fd2b4e514cacb6fcbf4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=995e2a7929d545f8b60371f9db40f0d4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:10:54.636763 12546 tablet_service.cc:1467] Processing CreateTablet for tablet c1bcdbe5695a4fd2b4e514cacb6fcbf4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=995e2a7929d545f8b60371f9db40f0d4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:10:54.637063 12267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1bcdbe5695a4fd2b4e514cacb6fcbf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:54.637101 12407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1bcdbe5695a4fd2b4e514cacb6fcbf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:54.637276 12546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1bcdbe5695a4fd2b4e514cacb6fcbf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:54.644587 12637 tablet_bootstrap.cc:492] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P 6e5ea13b58b14b078a2cb8470cc95774: Bootstrap starting.
I20260620 23:10:54.644660 12635 tablet_bootstrap.cc:492] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P 01dc46e565e4425992677fd968c5cc93: Bootstrap starting.
I20260620 23:10:54.646078 12635 tablet_bootstrap.cc:654] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P 01dc46e565e4425992677fd968c5cc93: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:54.646296 12637 tablet_bootstrap.cc:654] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P 6e5ea13b58b14b078a2cb8470cc95774: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:54.647068 12636 tablet_bootstrap.cc:492] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P ef34c1deb4ab41e1890a653d8d8e659b: Bootstrap starting.
I20260620 23:10:54.648660 12636 tablet_bootstrap.cc:654] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P ef34c1deb4ab41e1890a653d8d8e659b: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:54.648891 12635 tablet_bootstrap.cc:492] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P 01dc46e565e4425992677fd968c5cc93: No bootstrap required, opened a new log
I20260620 23:10:54.649024 12635 ts_tablet_manager.cc:1397] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P 01dc46e565e4425992677fd968c5cc93: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 23:10:54.649682 12635 raft_consensus.cc:348] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P 01dc46e565e4425992677fd968c5cc93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39791 } } peers { permanent_uuid: "01dc46e565e4425992677fd968c5cc93" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45739 } } peers { permanent_uuid: "6e5ea13b58b14b078a2cb8470cc95774" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39095 } }
I20260620 23:10:54.649972 12635 raft_consensus.cc:374] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P 01dc46e565e4425992677fd968c5cc93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:10:54.650066 12635 raft_consensus.cc:729] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P 01dc46e565e4425992677fd968c5cc93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01dc46e565e4425992677fd968c5cc93, State: Initialized, Role: FOLLOWER
I20260620 23:10:54.650944 12637 tablet_bootstrap.cc:492] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P 6e5ea13b58b14b078a2cb8470cc95774: No bootstrap required, opened a new log
I20260620 23:10:54.651186 12637 ts_tablet_manager.cc:1397] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P 6e5ea13b58b14b078a2cb8470cc95774: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260620 23:10:54.651224 12635 consensus_queue.cc:260] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P 01dc46e565e4425992677fd968c5cc93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39791 } } peers { permanent_uuid: "01dc46e565e4425992677fd968c5cc93" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45739 } } peers { permanent_uuid: "6e5ea13b58b14b078a2cb8470cc95774" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39095 } }
I20260620 23:10:54.651655 12656 tablet_replica.cc:439] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P 01dc46e565e4425992677fd968c5cc93: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39791 } } peers { permanent_uuid: "01dc46e565e4425992677fd968c5cc93" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45739 } } peers { permanent_uuid: "6e5ea13b58b14b078a2cb8470cc95774" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39095 } } }
I20260620 23:10:54.651963 12656 tablet_replica.cc:442] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P 01dc46e565e4425992677fd968c5cc93: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:10:54.651949 12637 raft_consensus.cc:348] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P 6e5ea13b58b14b078a2cb8470cc95774 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39791 } } peers { permanent_uuid: "01dc46e565e4425992677fd968c5cc93" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45739 } } peers { permanent_uuid: "6e5ea13b58b14b078a2cb8470cc95774" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39095 } }
I20260620 23:10:54.652191 12637 raft_consensus.cc:374] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P 6e5ea13b58b14b078a2cb8470cc95774 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:10:54.652275 12637 raft_consensus.cc:729] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P 6e5ea13b58b14b078a2cb8470cc95774 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e5ea13b58b14b078a2cb8470cc95774, State: Initialized, Role: FOLLOWER
I20260620 23:10:54.652520 12637 consensus_queue.cc:260] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P 6e5ea13b58b14b078a2cb8470cc95774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39791 } } peers { permanent_uuid: "01dc46e565e4425992677fd968c5cc93" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45739 } } peers { permanent_uuid: "6e5ea13b58b14b078a2cb8470cc95774" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39095 } }
I20260620 23:10:54.652932 12642 tablet_replica.cc:439] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P 6e5ea13b58b14b078a2cb8470cc95774: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39791 } } peers { permanent_uuid: "01dc46e565e4425992677fd968c5cc93" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45739 } } peers { permanent_uuid: "6e5ea13b58b14b078a2cb8470cc95774" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39095 } } }
I20260620 23:10:54.653209 12635 ts_tablet_manager.cc:1428] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P 01dc46e565e4425992677fd968c5cc93: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260620 23:10:54.653309 12642 tablet_replica.cc:442] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P 6e5ea13b58b14b078a2cb8470cc95774: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:10:54.654294 12637 ts_tablet_manager.cc:1428] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P 6e5ea13b58b14b078a2cb8470cc95774: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 23:10:54.661526 12636 tablet_bootstrap.cc:492] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P ef34c1deb4ab41e1890a653d8d8e659b: No bootstrap required, opened a new log
I20260620 23:10:54.661864 12636 ts_tablet_manager.cc:1397] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P ef34c1deb4ab41e1890a653d8d8e659b: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.000s
I20260620 23:10:54.662773 12636 raft_consensus.cc:348] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P ef34c1deb4ab41e1890a653d8d8e659b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39791 } } peers { permanent_uuid: "01dc46e565e4425992677fd968c5cc93" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45739 } } peers { permanent_uuid: "6e5ea13b58b14b078a2cb8470cc95774" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39095 } }
I20260620 23:10:54.663096 12636 raft_consensus.cc:374] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P ef34c1deb4ab41e1890a653d8d8e659b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:10:54.663183 12636 raft_consensus.cc:729] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P ef34c1deb4ab41e1890a653d8d8e659b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef34c1deb4ab41e1890a653d8d8e659b, State: Initialized, Role: FOLLOWER
I20260620 23:10:54.663448 12636 consensus_queue.cc:260] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P ef34c1deb4ab41e1890a653d8d8e659b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39791 } } peers { permanent_uuid: "01dc46e565e4425992677fd968c5cc93" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45739 } } peers { permanent_uuid: "6e5ea13b58b14b078a2cb8470cc95774" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39095 } }
I20260620 23:10:54.664009 12636 ts_tablet_manager.cc:1428] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P ef34c1deb4ab41e1890a653d8d8e659b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:10:54.667109 12643 tablet_replica.cc:439] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P ef34c1deb4ab41e1890a653d8d8e659b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39791 } } peers { permanent_uuid: "01dc46e565e4425992677fd968c5cc93" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45739 } } peers { permanent_uuid: "6e5ea13b58b14b078a2cb8470cc95774" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39095 } } }
I20260620 23:10:54.667481 12643 tablet_replica.cc:442] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P ef34c1deb4ab41e1890a653d8d8e659b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:10:54.730017 12643 raft_consensus.cc:482] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P ef34c1deb4ab41e1890a653d8d8e659b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:10:54.730248 12643 raft_consensus.cc:504] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P ef34c1deb4ab41e1890a653d8d8e659b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39791 } } peers { permanent_uuid: "01dc46e565e4425992677fd968c5cc93" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45739 } } peers { permanent_uuid: "6e5ea13b58b14b078a2cb8470cc95774" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39095 } }
I20260620 23:10:54.732156 12643 leader_election.cc:288] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P ef34c1deb4ab41e1890a653d8d8e659b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01dc46e565e4425992677fd968c5cc93 (127.10.191.194:45739), 6e5ea13b58b14b078a2cb8470cc95774 (127.10.191.195:39095)
I20260620 23:10:54.742764 12634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:54.732942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:50227 (local address 127.10.191.194:45739)
0620 23:10:54.733127 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:54.733134 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:54.735488 (+ 2354us) server_negotiation.cc:407] Connection header received
0620 23:10:54.735680 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:54.735690 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:54.735794 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:54.735898 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:10:54.736774 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.737732 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.740331 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.741698 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.741879 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:54.741976 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:54.742289 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 23:10:54.742422 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 23:10:54.743047 12632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:54.733031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:53005 (local address 127.10.191.195:39095)
0620 23:10:54.733287 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:54.733293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:54.736325 (+ 3032us) server_negotiation.cc:407] Connection header received
0620 23:10:54.736578 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:54.736583 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:54.736653 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:54.736853 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:10:54.737714 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.738443 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.740843 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.742081 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.742231 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:54.742308 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:54.742622 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 23:10:54.742769 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260620 23:10:54.743114 12676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:54.732542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.195:39095 (local address 127.10.191.193:53005)
0620 23:10:54.736226 (+ 3684us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:54.736247 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:54.736425 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:54.737105 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:54.737110 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:54.737135 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:10:54.737488 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:54.737500 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.738582 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.738588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:54.740453 (+ 1865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:54.740467 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.742295 (+ 1828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.742302 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:54.742452 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:54.742481 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 23:10:54.742573 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 23:10:54.742706 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3554,"thread_start_us":3507,"threads_started":1}
I20260620 23:10:54.743714 12571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1bcdbe5695a4fd2b4e514cacb6fcbf4" candidate_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e5ea13b58b14b078a2cb8470cc95774" is_pre_election: true
I20260620 23:10:54.743997 12571 raft_consensus.cc:2454] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P 6e5ea13b58b14b078a2cb8470cc95774 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef34c1deb4ab41e1890a653d8d8e659b in term 0.
I20260620 23:10:54.744293 12675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:54.732503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.194:45739 (local address 127.10.191.193:50227)
0620 23:10:54.735344 (+ 2841us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:54.735366 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:54.735521 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:54.736080 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:54.736084 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:54.736114 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:10:54.736590 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:54.736603 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.737870 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.737875 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:54.740140 (+ 2265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:54.740155 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.741901 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.741906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:54.742071 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:54.742107 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 23:10:54.743755 (+ 1648us) client_negotiation.cc:240] Negotiation successful
0620 23:10:54.743935 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2709,"thread_start_us":2549,"threads_started":1}
I20260620 23:10:54.744483 12219 leader_election.cc:302] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P ef34c1deb4ab41e1890a653d8d8e659b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e5ea13b58b14b078a2cb8470cc95774, ef34c1deb4ab41e1890a653d8d8e659b; no voters:
I20260620 23:10:54.744941 12643 raft_consensus.cc:2781] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P ef34c1deb4ab41e1890a653d8d8e659b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:10:54.745072 12643 raft_consensus.cc:482] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P ef34c1deb4ab41e1890a653d8d8e659b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:10:54.745172 12643 raft_consensus.cc:3037] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P ef34c1deb4ab41e1890a653d8d8e659b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:10:54.745066 12427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1bcdbe5695a4fd2b4e514cacb6fcbf4" candidate_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01dc46e565e4425992677fd968c5cc93" is_pre_election: true
I20260620 23:10:54.745532 12427 raft_consensus.cc:2454] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P 01dc46e565e4425992677fd968c5cc93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef34c1deb4ab41e1890a653d8d8e659b in term 0.
I20260620 23:10:54.747584 12643 raft_consensus.cc:504] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P ef34c1deb4ab41e1890a653d8d8e659b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39791 } } peers { permanent_uuid: "01dc46e565e4425992677fd968c5cc93" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45739 } } peers { permanent_uuid: "6e5ea13b58b14b078a2cb8470cc95774" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39095 } }
I20260620 23:10:54.748226 12643 leader_election.cc:288] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P ef34c1deb4ab41e1890a653d8d8e659b [CANDIDATE]: Term 1 election: Requested vote from peers 01dc46e565e4425992677fd968c5cc93 (127.10.191.194:45739), 6e5ea13b58b14b078a2cb8470cc95774 (127.10.191.195:39095)
I20260620 23:10:54.748629 12427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1bcdbe5695a4fd2b4e514cacb6fcbf4" candidate_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01dc46e565e4425992677fd968c5cc93"
I20260620 23:10:54.748850 12427 raft_consensus.cc:3037] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P 01dc46e565e4425992677fd968c5cc93 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:10:54.748845 12571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1bcdbe5695a4fd2b4e514cacb6fcbf4" candidate_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e5ea13b58b14b078a2cb8470cc95774"
I20260620 23:10:54.749076 12571 raft_consensus.cc:3037] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P 6e5ea13b58b14b078a2cb8470cc95774 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:10:54.751976 12571 raft_consensus.cc:2454] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P 6e5ea13b58b14b078a2cb8470cc95774 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef34c1deb4ab41e1890a653d8d8e659b in term 1.
I20260620 23:10:54.752295 12427 raft_consensus.cc:2454] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P 01dc46e565e4425992677fd968c5cc93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef34c1deb4ab41e1890a653d8d8e659b in term 1.
I20260620 23:10:54.752353 12219 leader_election.cc:302] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P ef34c1deb4ab41e1890a653d8d8e659b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e5ea13b58b14b078a2cb8470cc95774, ef34c1deb4ab41e1890a653d8d8e659b; no voters:
I20260620 23:10:54.752667 12643 raft_consensus.cc:2781] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P ef34c1deb4ab41e1890a653d8d8e659b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:10:54.753170 12643 raft_consensus.cc:686] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P ef34c1deb4ab41e1890a653d8d8e659b [term 1 LEADER]: Becoming Leader. State: Replica: ef34c1deb4ab41e1890a653d8d8e659b, State: Running, Role: LEADER
I20260620 23:10:54.753579 12643 consensus_queue.cc:237] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P ef34c1deb4ab41e1890a653d8d8e659b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39791 } } peers { permanent_uuid: "01dc46e565e4425992677fd968c5cc93" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45739 } } peers { permanent_uuid: "6e5ea13b58b14b078a2cb8470cc95774" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39095 } }
I20260620 23:10:54.756573 12677 tablet_replica.cc:439] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P ef34c1deb4ab41e1890a653d8d8e659b: TxnStatusTablet state changed. Reason: New leader ef34c1deb4ab41e1890a653d8d8e659b. Latest consensus state: current_term: 1 leader_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39791 } } peers { permanent_uuid: "01dc46e565e4425992677fd968c5cc93" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45739 } } peers { permanent_uuid: "6e5ea13b58b14b078a2cb8470cc95774" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39095 } } }
I20260620 23:10:54.756893 12677 tablet_replica.cc:442] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P ef34c1deb4ab41e1890a653d8d8e659b: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:10:54.757764 12679 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 23:10:54.757076 12131 catalog_manager.cc:5495] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P ef34c1deb4ab41e1890a653d8d8e659b reported cstate change: term changed from 0 to 1, leader changed from <none> to ef34c1deb4ab41e1890a653d8d8e659b (127.10.191.193). New cstate: current_term: 1 leader_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01dc46e565e4425992677fd968c5cc93" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e5ea13b58b14b078a2cb8470cc95774" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39095 } health_report { overall_health: UNKNOWN } } }
I20260620 23:10:54.758318 12679 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 23:10:54.795995 12671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:54.788741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:39791 (local address 127.0.0.1:47730)
0620 23:10:54.788980 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:54.789002 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:54.789171 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:54.789699 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:54.789706 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:54.789719 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:10:54.789898 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:54.789908 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.791450 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.791457 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:54.793102 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:54.793116 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.794078 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.794085 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:54.794261 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:54.794333 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:10:54.795413 (+ 1080us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:10:54.795435 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 23:10:54.795528 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 23:10:54.795692 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 23:10:54.795993 12631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:54.788636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47730 (local address 127.10.191.193:39791)
0620 23:10:54.788837 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:54.788845 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:54.789084 (+ 239us) server_negotiation.cc:407] Connection header received
0620 23:10:54.789389 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:54.789395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:54.789504 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:54.789657 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:10:54.790038 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.791295 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.793249 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.793918 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.794054 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:54.794542 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:10:54.795283 (+ 741us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:10:54.795454 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:54.795563 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 23:10:54.795654 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:10:54.800585 12571 raft_consensus.cc:1261] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P 6e5ea13b58b14b078a2cb8470cc95774 [term 1 FOLLOWER]: Refusing update from remote peer ef34c1deb4ab41e1890a653d8d8e659b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:10:54.800938 12427 raft_consensus.cc:1261] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P 01dc46e565e4425992677fd968c5cc93 [term 1 FOLLOWER]: Refusing update from remote peer ef34c1deb4ab41e1890a653d8d8e659b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:10:54.801635 12643 consensus_queue.cc:1035] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P ef34c1deb4ab41e1890a653d8d8e659b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e5ea13b58b14b078a2cb8470cc95774" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:10:54.802268 12643 consensus_queue.cc:1035] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P ef34c1deb4ab41e1890a653d8d8e659b [LEADER]: Connected to new peer: Peer: permanent_uuid: "01dc46e565e4425992677fd968c5cc93" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:10:54.815461 12656 tablet_replica.cc:439] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P 01dc46e565e4425992677fd968c5cc93: TxnStatusTablet state changed. Reason: New leader ef34c1deb4ab41e1890a653d8d8e659b. Latest consensus state: current_term: 1 leader_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39791 } } peers { permanent_uuid: "01dc46e565e4425992677fd968c5cc93" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45739 } } peers { permanent_uuid: "6e5ea13b58b14b078a2cb8470cc95774" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39095 } } }
I20260620 23:10:54.815794 12656 tablet_replica.cc:442] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P 01dc46e565e4425992677fd968c5cc93: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:10:54.817530 12677 tablet_replica.cc:439] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P ef34c1deb4ab41e1890a653d8d8e659b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39791 } } peers { permanent_uuid: "01dc46e565e4425992677fd968c5cc93" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45739 } } peers { permanent_uuid: "6e5ea13b58b14b078a2cb8470cc95774" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39095 } } }
I20260620 23:10:54.817771 12677 tablet_replica.cc:442] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P ef34c1deb4ab41e1890a653d8d8e659b: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:10:54.819316 12649 mvcc.cc:205] Tried to move back new op lower bound from 7299059936455888896 to 7299059936274452480. Current Snapshot: MvccSnapshot[applied={T|T < 7299059936455888896}]
I20260620 23:10:54.820578 12656 tablet_replica.cc:439] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P 01dc46e565e4425992677fd968c5cc93: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39791 } } peers { permanent_uuid: "01dc46e565e4425992677fd968c5cc93" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45739 } } peers { permanent_uuid: "6e5ea13b58b14b078a2cb8470cc95774" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39095 } } }
I20260620 23:10:54.820847 12656 tablet_replica.cc:442] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P 01dc46e565e4425992677fd968c5cc93: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:10:54.821262 12642 tablet_replica.cc:439] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P 6e5ea13b58b14b078a2cb8470cc95774: TxnStatusTablet state changed. Reason: New leader ef34c1deb4ab41e1890a653d8d8e659b. Latest consensus state: current_term: 1 leader_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39791 } } peers { permanent_uuid: "01dc46e565e4425992677fd968c5cc93" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45739 } } peers { permanent_uuid: "6e5ea13b58b14b078a2cb8470cc95774" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39095 } } }
I20260620 23:10:54.821496 12642 tablet_replica.cc:442] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P 6e5ea13b58b14b078a2cb8470cc95774: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:10:54.826205 12677 tablet_replica.cc:439] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P ef34c1deb4ab41e1890a653d8d8e659b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39791 } } peers { permanent_uuid: "01dc46e565e4425992677fd968c5cc93" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45739 } } peers { permanent_uuid: "6e5ea13b58b14b078a2cb8470cc95774" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39095 } } }
I20260620 23:10:54.826455 12677 tablet_replica.cc:442] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P ef34c1deb4ab41e1890a653d8d8e659b: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:10:54.833238 12642 tablet_replica.cc:439] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P 6e5ea13b58b14b078a2cb8470cc95774: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39791 } } peers { permanent_uuid: "01dc46e565e4425992677fd968c5cc93" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45739 } } peers { permanent_uuid: "6e5ea13b58b14b078a2cb8470cc95774" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39095 } } }
I20260620 23:10:54.833490 12642 tablet_replica.cc:442] T c1bcdbe5695a4fd2b4e514cacb6fcbf4 P 6e5ea13b58b14b078a2cb8470cc95774: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:10:54.849519 12631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:54.841738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:41625 (local address 127.10.191.193:39791)
0620 23:10:54.841923 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:54.841930 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:54.842581 (+ 651us) server_negotiation.cc:407] Connection header received
0620 23:10:54.842819 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:54.842826 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:54.842947 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:54.843046 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:10:54.843643 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.844841 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.847133 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.847860 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.847995 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:54.848687 (+ 692us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:10:54.848823 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:10:54.848917 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:54.849113 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 23:10:54.849195 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 23:10:54.849640 12686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:54.841630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:39791 (local address 127.10.191.194:41625)
0620 23:10:54.842461 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:54.842481 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:54.842643 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:54.843078 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:54.843082 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:54.843097 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:10:54.843471 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:54.843483 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.844986 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.844992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:54.846981 (+ 1989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:54.846998 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.848068 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.848080 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:54.848298 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:54.848385 (+ 87us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:10:54.848969 (+ 584us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:10:54.848987 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 23:10:54.849086 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 23:10:54.849244 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":464,"threads_started":1}
I20260620 23:10:54.862422 12288 raft_consensus.cc:1261] T a052bbcd931449279f88723c2bf6f723 P ef34c1deb4ab41e1890a653d8d8e659b [term 1 FOLLOWER]: Refusing update from remote peer 01dc46e565e4425992677fd968c5cc93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:10:54.862442 12571 raft_consensus.cc:1261] T a052bbcd931449279f88723c2bf6f723 P 6e5ea13b58b14b078a2cb8470cc95774 [term 1 FOLLOWER]: Refusing update from remote peer 01dc46e565e4425992677fd968c5cc93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:10:54.863059 12641 consensus_queue.cc:1035] T a052bbcd931449279f88723c2bf6f723 P 01dc46e565e4425992677fd968c5cc93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e5ea13b58b14b078a2cb8470cc95774" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:10:54.863513 12656 consensus_queue.cc:1035] T a052bbcd931449279f88723c2bf6f723 P 01dc46e565e4425992677fd968c5cc93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef34c1deb4ab41e1890a653d8d8e659b" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:10:54.931919 12634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:54.920639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:38117 (local address 127.10.191.194:45739)
0620 23:10:54.920855 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:54.920862 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:54.922730 (+ 1868us) server_negotiation.cc:407] Connection header received
0620 23:10:54.923063 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:54.923070 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:54.923163 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:54.923261 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:10:54.925980 (+ 2719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.927215 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.929387 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.930172 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.930273 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:54.931040 (+ 767us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:10:54.931207 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:10:54.931305 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:54.931474 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 23:10:54.931559 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 23:10:54.932255 12692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:54.921012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.194:45739 (local address 127.10.191.193:38117)
0620 23:10:54.922591 (+ 1579us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:54.922611 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:54.922799 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:54.923304 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:54.923309 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:54.923323 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:10:54.925740 (+ 2417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:54.925756 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.927355 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.927362 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:54.929202 (+ 1840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:54.929219 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.930492 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.930499 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:54.930706 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:54.930784 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:10:54.931316 (+ 532us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:10:54.931339 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 23:10:54.931426 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 23:10:54.931583 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1343,"thread_start_us":1199,"threads_started":1}
I20260620 23:10:54.973253 12132 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:10:54.973510 12132 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:10:54.976845 12132 catalog_manager.cc:5782] T 00000000000000000000000000000000 P af70cc3d88304030b28e33c149ca3aca: Sending DeleteTablet for 3 replicas of tablet a052bbcd931449279f88723c2bf6f723
I20260620 23:10:54.977876 12407 tablet_service.cc:1514] Processing DeleteTablet for tablet a052bbcd931449279f88723c2bf6f723 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:10:54 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:43356
I20260620 23:10:54.978163 12661 tablet_replica.cc:330] T a052bbcd931449279f88723c2bf6f723 P 01dc46e565e4425992677fd968c5cc93: stopping tablet replica
I20260620 23:10:54.978341 12661 raft_consensus.cc:2229] T a052bbcd931449279f88723c2bf6f723 P 01dc46e565e4425992677fd968c5cc93 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:10:54.978477 12546 tablet_service.cc:1514] Processing DeleteTablet for tablet a052bbcd931449279f88723c2bf6f723 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:10:54 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:40676
I20260620 23:10:54.978673 12661 raft_consensus.cc:2258] T a052bbcd931449279f88723c2bf6f723 P 01dc46e565e4425992677fd968c5cc93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:10:54.978729 12659 tablet_replica.cc:330] T a052bbcd931449279f88723c2bf6f723 P 6e5ea13b58b14b078a2cb8470cc95774: stopping tablet replica
I20260620 23:10:54.978945 12659 raft_consensus.cc:2229] T a052bbcd931449279f88723c2bf6f723 P 6e5ea13b58b14b078a2cb8470cc95774 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:10:54.979122 12659 raft_consensus.cc:2258] T a052bbcd931449279f88723c2bf6f723 P 6e5ea13b58b14b078a2cb8470cc95774 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:10:54.979727 12267 tablet_service.cc:1514] Processing DeleteTablet for tablet a052bbcd931449279f88723c2bf6f723 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:10:54 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:47716
I20260620 23:10:54.979928 12661 ts_tablet_manager.cc:1905] T a052bbcd931449279f88723c2bf6f723 P 01dc46e565e4425992677fd968c5cc93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:10:54.980314 12659 ts_tablet_manager.cc:1905] T a052bbcd931449279f88723c2bf6f723 P 6e5ea13b58b14b078a2cb8470cc95774: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:10:54.983065 12660 tablet_replica.cc:330] T a052bbcd931449279f88723c2bf6f723 P ef34c1deb4ab41e1890a653d8d8e659b: stopping tablet replica
I20260620 23:10:54.983274 12660 raft_consensus.cc:2229] T a052bbcd931449279f88723c2bf6f723 P ef34c1deb4ab41e1890a653d8d8e659b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:10:54.983469 12660 raft_consensus.cc:2258] T a052bbcd931449279f88723c2bf6f723 P ef34c1deb4ab41e1890a653d8d8e659b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:10:54.984707 12660 ts_tablet_manager.cc:1905] T a052bbcd931449279f88723c2bf6f723 P ef34c1deb4ab41e1890a653d8d8e659b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:10:54.987825 12661 ts_tablet_manager.cc:1918] T a052bbcd931449279f88723c2bf6f723 P 01dc46e565e4425992677fd968c5cc93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:10:54.988032 12661 log.cc:1196] T a052bbcd931449279f88723c2bf6f723 P 01dc46e565e4425992677fd968c5cc93: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-1/wal/wals/a052bbcd931449279f88723c2bf6f723
I20260620 23:10:54.988600 12661 ts_tablet_manager.cc:1939] T a052bbcd931449279f88723c2bf6f723 P 01dc46e565e4425992677fd968c5cc93: Deleting consensus metadata
I20260620 23:10:54.989987 12119 catalog_manager.cc:4841] TS 01dc46e565e4425992677fd968c5cc93 (127.10.191.194:45739): tablet a052bbcd931449279f88723c2bf6f723 (table test-table [id=53a312f4fa934432bd1416ff50b4738d]) successfully deleted
I20260620 23:10:54.992070 12660 ts_tablet_manager.cc:1918] T a052bbcd931449279f88723c2bf6f723 P ef34c1deb4ab41e1890a653d8d8e659b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:10:54.992231 12660 log.cc:1196] T a052bbcd931449279f88723c2bf6f723 P ef34c1deb4ab41e1890a653d8d8e659b: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-0/wal/wals/a052bbcd931449279f88723c2bf6f723
I20260620 23:10:54.994091 12659 ts_tablet_manager.cc:1918] T a052bbcd931449279f88723c2bf6f723 P 6e5ea13b58b14b078a2cb8470cc95774: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:10:54.994284 12659 log.cc:1196] T a052bbcd931449279f88723c2bf6f723 P 6e5ea13b58b14b078a2cb8470cc95774: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781997042893763-11007-0/minicluster-data/ts-2/wal/wals/a052bbcd931449279f88723c2bf6f723
I20260620 23:10:54.994676 12659 ts_tablet_manager.cc:1939] T a052bbcd931449279f88723c2bf6f723 P 6e5ea13b58b14b078a2cb8470cc95774: Deleting consensus metadata
I20260620 23:10:54.995777 12116 catalog_manager.cc:4841] TS 6e5ea13b58b14b078a2cb8470cc95774 (127.10.191.195:39095): tablet a052bbcd931449279f88723c2bf6f723 (table test-table [id=53a312f4fa934432bd1416ff50b4738d]) successfully deleted
I20260620 23:10:54.996258 12660 ts_tablet_manager.cc:1939] T a052bbcd931449279f88723c2bf6f723 P ef34c1deb4ab41e1890a653d8d8e659b: Deleting consensus metadata
I20260620 23:10:54.997355 12119 catalog_manager.cc:4841] TS ef34c1deb4ab41e1890a653d8d8e659b (127.10.191.193:39791): tablet a052bbcd931449279f88723c2bf6f723 (table test-table [id=53a312f4fa934432bd1416ff50b4738d]) successfully deleted
Jun 20 23:10:55 dist-test-slave-jkbq krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997054, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.191.193@KRBTEST.COM
I20260620 23:10:55.016121 12631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:54.984573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47742 (local address 127.10.191.193:39791)
0620 23:10:54.984753 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:54.984759 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:54.991521 (+ 6762us) server_negotiation.cc:407] Connection header received
0620 23:10:54.991904 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:54.991910 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:54.991987 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:54.992108 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:54.994097 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.995252 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.997447 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:54.998197 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:54.998303 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:55.008562 (+ 10259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:55.008581 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:55.008602 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:55.008660 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:55.012647 (+ 3987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:55.013540 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:55.013550 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:55.013554 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:55.013747 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:55.014170 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:55.014175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:55.014179 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:55.014667 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:55.015058 (+ 391us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:55.015294 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 23:10:55.015538 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 23:10:55.016919 12333 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47742
I20260620 23:10:55.042222 12180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:55.020128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49578 (local address 127.10.191.254:42087)
0620 23:10:55.020325 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:55.020331 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:55.026677 (+ 6346us) server_negotiation.cc:407] Connection header received
0620 23:10:55.026901 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:55.026907 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:55.026995 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:55.027071 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:55.028918 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:55.030084 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:55.031951 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:55.032658 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:55.032787 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:55.035635 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:55.035643 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:55.035648 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:55.035689 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:55.039150 (+ 3461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:55.039969 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:55.039975 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:55.039979 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:55.040126 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:55.040538 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:55.040564 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:55.040567 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:55.040987 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:55.041226 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:55.041632 (+ 406us) server_negotiation.cc:299] Negotiation successful
0620 23:10:55.041865 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 23:10:55.042776 12132 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49578
I20260620 23:10:55.046770 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 12203
I20260620 23:10:55.063866 12631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:55.047761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47758 (local address 127.10.191.193:39791)
0620 23:10:55.047967 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:55.047976 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:55.048404 (+ 428us) server_negotiation.cc:407] Connection header received
0620 23:10:55.048582 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:55.048588 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:55.048672 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:55.048778 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:55.050199 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:55.051283 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:55.053221 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:55.053841 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:55.053978 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:55.057039 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:55.057048 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:55.057054 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:55.057108 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:55.060909 (+ 3801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:55.061716 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:55.061721 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:55.061725 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:55.061868 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:55.062277 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:55.062282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:55.062285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:55.062658 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:55.062943 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:55.063258 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 23:10:55.063514 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 23:10:55.064445 12333 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47758
W20260620 23:10:55.065035 11007 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 12203: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:10:55.065225 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 12203
I20260620 23:10:55.081399 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 12342
Jun 20 23:10:55 dist-test-slave-jkbq krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997054, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.191.194@KRBTEST.COM
I20260620 23:10:55.095777 12634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:55.082166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43368 (local address 127.10.191.194:45739)
0620 23:10:55.082354 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:55.082361 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:55.082382 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:10:55.082592 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:55.082598 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:55.082668 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:55.082793 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:55.084086 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:55.084836 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:55.086216 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:55.086656 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:55.086789 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:55.090665 (+ 3876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:55.090676 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:55.090691 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:55.090721 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:55.093556 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:55.094128 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:55.094135 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:55.094137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:55.094254 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:55.094551 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:55.094554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:55.094556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:55.094922 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:55.095194 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:55.095395 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 23:10:55.095549 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 23:10:55.096470 12472 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43368
W20260620 23:10:55.097062 11007 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 12342: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:10:55.097239 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 12342
I20260620 23:10:55.117950 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 12481
Jun 20 23:10:55 dist-test-slave-jkbq krb5kdc[12087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997054, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.191.195@KRBTEST.COM
I20260620 23:10:55.132275 12632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:55.118716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40682 (local address 127.10.191.195:39095)
0620 23:10:55.118966 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:55.118970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:55.118989 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:10:55.119142 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:55.119146 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:55.119205 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:55.119294 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:55.120519 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:55.121207 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:55.122393 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:55.122802 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:55.123033 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:55.127106 (+ 4073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:55.127120 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:55.127136 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:55.127171 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:55.129710 (+ 2539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:55.130469 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:55.130476 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:55.130478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:55.130586 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:55.130939 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:55.130946 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:55.130952 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:55.131310 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:55.131519 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:55.131764 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 23:10:55.131938 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 23:10:55.132967 12611 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40682
W20260620 23:10:55.133574 11007 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 12481: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:10:55.133729 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 12481
I20260620 23:10:55.150373 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 12100
I20260620 23:10:55.151518 12172 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:10:55.264756 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 12100
2026-06-20T23:10:55Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4348 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:10:55 dist-test-slave-jkbq krb5kdc[12702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:10:55 dist-test-slave-jkbq krb5kdc[12702](info): set up 2 sockets
Jun 20 23:10:55 dist-test-slave-jkbq krb5kdc[12702](info): commencing operation
krb5kdc: starting...
W20260620 23:10:56.350999 11007 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.005s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Jun 20 23:10:56 dist-test-slave-jkbq krb5kdc[12702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:10:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:10:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.191.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
I20260620 23:10:56.528805 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:36523
--webserver_interface=127.10.191.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:44933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:36523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:10:56.835494 12715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:10:56.835772 12715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:10:56.835963 12715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:10:56.843068 12715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:10:56.843168 12715 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 23:10:56.843263 12715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:10:56.843375 12715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:10:56.843438 12715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:10:56.843542 12715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:10:56.853276 12715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:44933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:36523
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:36523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:10:56.854588 12715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:10:56.855998 12715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:10:56.868297 12723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:56.868346 12721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:56.868564 12720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:10:56.869107 12715 server_base.cc:1034] running on GCE node
I20260620 23:10:56.869933 12715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:10:56.871346 12715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:10:56.872696 12715 hybrid_clock.cc:648] HybridClock initialized: now 1781997056872633 us; error 35 us; skew 500 ppm
I20260620 23:10:56.873061 12715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:10:56 dist-test-slave-jkbq krb5kdc[12702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997056, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:10:56.876055 12715 init.cc:377] Logged in from keytab as kudu/127.10.191.254@KRBTEST.COM (short username kudu)
I20260620 23:10:56.877405 12715 webserver.cc:460] Webserver started at http://127.10.191.254:36923/ using document root <none> and password file <none>
I20260620 23:10:56.878104 12715 fs_manager.cc:359] Metadata directory not provided
I20260620 23:10:56.878237 12715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:10:56.878551 12715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:10:56.881425 12715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "69ad19c2711d4c08b9b99bd0fe396704"
format_stamp: "Formatted at 2026-06-20 23:10:56 on dist-test-slave-jkbq"
server_key: "a98fbbe6c0014e9355521fd296803d2f"
server_key_iv: "22ba3bd4ea3f08f9441926cb6315ff4d"
server_key_version: "encryptionkey@0"
I20260620 23:10:56.882186 12715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "69ad19c2711d4c08b9b99bd0fe396704"
format_stamp: "Formatted at 2026-06-20 23:10:56 on dist-test-slave-jkbq"
server_key: "a98fbbe6c0014e9355521fd296803d2f"
server_key_iv: "22ba3bd4ea3f08f9441926cb6315ff4d"
server_key_version: "encryptionkey@0"
I20260620 23:10:56.887269 12715 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 23:10:56.890522 12730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:56.891848 12715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:10:56.892033 12715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "69ad19c2711d4c08b9b99bd0fe396704"
format_stamp: "Formatted at 2026-06-20 23:10:56 on dist-test-slave-jkbq"
server_key: "a98fbbe6c0014e9355521fd296803d2f"
server_key_iv: "22ba3bd4ea3f08f9441926cb6315ff4d"
server_key_version: "encryptionkey@0"
I20260620 23:10:56.892246 12715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:10:56.941126 12715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:10:56.945385 12715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:10:56.945675 12715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:10:56.967726 12715 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:36523
I20260620 23:10:56.967749 12791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:36523 every 8 connection(s)
I20260620 23:10:56.969120 12715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:10:56.972785 12792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:56.974794 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 12715
I20260620 23:10:56.975024 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:10:56.975689 11007 external_mini_cluster.cc:1428] Setting key 83a591ccea2b64b97f7835f8bcaa1705
I20260620 23:10:56.984566 12792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69ad19c2711d4c08b9b99bd0fe396704: Bootstrap starting.
I20260620 23:10:56.988296 12792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69ad19c2711d4c08b9b99bd0fe396704: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:56.989626 12792 log.cc:824] T 00000000000000000000000000000000 P 69ad19c2711d4c08b9b99bd0fe396704: Log is configured to *not* fsync() on all Append() calls
Jun 20 23:10:56 dist-test-slave-jkbq krb5kdc[12702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:10:56.992604 12792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69ad19c2711d4c08b9b99bd0fe396704: No bootstrap required, opened a new log
I20260620 23:10:56.998591 12795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:56.977732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58162 (local address 127.10.191.254:36523)
0620 23:10:56.978557 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:56.978569 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:56.978610 (+ 41us) server_negotiation.cc:407] Connection header received
0620 23:10:56.980287 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:56.980308 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:56.980694 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:56.981027 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:56.982739 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:56.984057 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:56.985769 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:56.986443 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:56.986573 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:56.991360 (+ 4787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:56.991367 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:56.991381 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:56.991412 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:56.994439 (+ 3027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:56.995302 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:56.995307 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:56.995313 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:56.995476 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:56.995876 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:56.995879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:56.995881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:56.996469 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:56.996758 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:56.997157 (+ 399us) server_negotiation.cc:299] Negotiation successful
0620 23:10:56.997545 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":422,"threads_started":1}
I20260620 23:10:57.001255 12792 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69ad19c2711d4c08b9b99bd0fe396704 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69ad19c2711d4c08b9b99bd0fe396704" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 36523 } }
I20260620 23:10:57.001621 12792 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69ad19c2711d4c08b9b99bd0fe396704 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:10:57.001726 12792 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69ad19c2711d4c08b9b99bd0fe396704 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ad19c2711d4c08b9b99bd0fe396704, State: Initialized, Role: FOLLOWER
I20260620 23:10:57.002267 12792 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69ad19c2711d4c08b9b99bd0fe396704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69ad19c2711d4c08b9b99bd0fe396704" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 36523 } }
I20260620 23:10:57.002511 12792 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69ad19c2711d4c08b9b99bd0fe396704 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:10:57.002657 12792 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69ad19c2711d4c08b9b99bd0fe396704 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:10:57.002840 12792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69ad19c2711d4c08b9b99bd0fe396704 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:10:57.005107 12792 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69ad19c2711d4c08b9b99bd0fe396704 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69ad19c2711d4c08b9b99bd0fe396704" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 36523 } }
I20260620 23:10:57.005548 12792 leader_election.cc:302] T 00000000000000000000000000000000 P 69ad19c2711d4c08b9b99bd0fe396704 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69ad19c2711d4c08b9b99bd0fe396704; no voters:
I20260620 23:10:57.006029 12792 leader_election.cc:288] T 00000000000000000000000000000000 P 69ad19c2711d4c08b9b99bd0fe396704 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:10:57.006186 12797 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69ad19c2711d4c08b9b99bd0fe396704 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:10:57.006970 12797 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69ad19c2711d4c08b9b99bd0fe396704 [term 1 LEADER]: Becoming Leader. State: Replica: 69ad19c2711d4c08b9b99bd0fe396704, State: Running, Role: LEADER
I20260620 23:10:57.007524 12797 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69ad19c2711d4c08b9b99bd0fe396704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69ad19c2711d4c08b9b99bd0fe396704" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 36523 } }
I20260620 23:10:57.008087 12792 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69ad19c2711d4c08b9b99bd0fe396704 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:10:57.010346 12798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69ad19c2711d4c08b9b99bd0fe396704 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69ad19c2711d4c08b9b99bd0fe396704" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69ad19c2711d4c08b9b99bd0fe396704" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 36523 } } }
I20260620 23:10:57.010653 12798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69ad19c2711d4c08b9b99bd0fe396704 [sys.catalog]: This master's current role is: LEADER
I20260620 23:10:57.011509 12803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:10:57.011364 12799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69ad19c2711d4c08b9b99bd0fe396704 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69ad19c2711d4c08b9b99bd0fe396704. Latest consensus state: current_term: 1 leader_uuid: "69ad19c2711d4c08b9b99bd0fe396704" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69ad19c2711d4c08b9b99bd0fe396704" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 36523 } } }
I20260620 23:10:57.012606 12799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69ad19c2711d4c08b9b99bd0fe396704 [sys.catalog]: This master's current role is: LEADER
I20260620 23:10:57.017757 12803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:10:57.026962 12803 catalog_manager.cc:1347] Generated new cluster ID: b74a2b892fa240d39702317152b5b040
I20260620 23:10:57.027079 12803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:10:57.053853 12803 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:10:57.055332 12803 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:10:57.062139 12803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69ad19c2711d4c08b9b99bd0fe396704: Generated new TSK 0
I20260620 23:10:57.063251 12803 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.191.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab.
Entry for principal kudu/127.10.191.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab.
I20260620 23:10:57.120788 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.193:0
--local_ip_for_outbound_sockets=127.10.191.193
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:36523
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:44933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:10:57.461619 12818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:10:57.461917 12818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:10:57.462020 12818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:10:57.469396 12818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:10:57.469496 12818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:10:57.469578 12818 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 23:10:57.469630 12818 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:10:57.469699 12818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.193
I20260620 23:10:57.479221 12818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:44933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.191.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.191.254:36523
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.193
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:10:57.480336 12818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:10:57.481559 12818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:10:57.494210 12823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:57.494299 12826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:57.494165 12824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:10:57.495399 12818 server_base.cc:1034] running on GCE node
I20260620 23:10:57.496049 12818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:10:57.497073 12818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:10:57.498298 12818 hybrid_clock.cc:648] HybridClock initialized: now 1781997057498262 us; error 51 us; skew 500 ppm
I20260620 23:10:57.498623 12818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:10:57 dist-test-slave-jkbq krb5kdc[12702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997057, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:10:57.501880 12818 init.cc:377] Logged in from keytab as kudu/127.10.191.193@KRBTEST.COM (short username kudu)
I20260620 23:10:57.503399 12818 webserver.cc:460] Webserver started at http://127.10.191.193:40551/ using document root <none> and password file <none>
I20260620 23:10:57.504021 12818 fs_manager.cc:359] Metadata directory not provided
I20260620 23:10:57.504118 12818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:10:57.504374 12818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:10:57.507207 12818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-0/data/instance:
uuid: "a4532051e6c94424963f3862691fb2a4"
format_stamp: "Formatted at 2026-06-20 23:10:57 on dist-test-slave-jkbq"
server_key: "503fe637f065d805a97911980e649c8d"
server_key_iv: "c443c4b9dcdf71a3c4bac8534f243f84"
server_key_version: "encryptionkey@0"
I20260620 23:10:57.507896 12818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance:
uuid: "a4532051e6c94424963f3862691fb2a4"
format_stamp: "Formatted at 2026-06-20 23:10:57 on dist-test-slave-jkbq"
server_key: "503fe637f065d805a97911980e649c8d"
server_key_iv: "c443c4b9dcdf71a3c4bac8534f243f84"
server_key_version: "encryptionkey@0"
I20260620 23:10:57.513124 12818 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:10:57.516494 12833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:57.517755 12818 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:10:57.517939 12818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-0/wal
uuid: "a4532051e6c94424963f3862691fb2a4"
format_stamp: "Formatted at 2026-06-20 23:10:57 on dist-test-slave-jkbq"
server_key: "503fe637f065d805a97911980e649c8d"
server_key_iv: "c443c4b9dcdf71a3c4bac8534f243f84"
server_key_version: "encryptionkey@0"
I20260620 23:10:57.518136 12818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:10:57.544121 12818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:10:57.548202 12818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:10:57.548496 12818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:10:57.551250 12818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:10:57.551369 12818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:57.551496 12818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:10:57.551640 12818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Jun 20 23:10:57 dist-test-slave-jkbq krb5kdc[12702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997057, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:10:57.574846 12842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:57.551781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:49455 (local address 127.10.191.254:36523)
0620 23:10:57.552535 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:57.552540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:57.553463 (+ 923us) server_negotiation.cc:407] Connection header received
0620 23:10:57.555541 (+ 2078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:57.555547 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:57.555637 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:57.555762 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:57.557964 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:57.559031 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:57.561653 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:57.562411 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:57.562573 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:57.567343 (+ 4770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:57.567352 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:57.567358 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:57.567407 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:57.570836 (+ 3429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:57.572057 (+ 1221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:57.572063 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:57.572068 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:57.572217 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:57.572834 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:57.572838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:57.572842 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:57.573223 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:57.573378 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:57.574434 (+ 1056us) server_negotiation.cc:299] Negotiation successful
0620 23:10:57.574595 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":563,"threads_started":1}
I20260620 23:10:57.577237 12843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:57.552459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:36523 (local address 127.10.191.193:49455)
0620 23:10:57.553291 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:57.553338 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:57.555238 (+ 1900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:57.556012 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:57.556020 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:57.556784 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:10:57.557783 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:57.557815 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:57.559177 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:57.559183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:57.561454 (+ 2271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:57.561466 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:57.562748 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:57.562760 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:57.563036 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:57.564074 (+ 1038us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:10:57.564113 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:10:57.567086 (+ 2973us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:10:57.571042 (+ 3956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:57.571055 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:57.571077 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:57.571906 (+ 829us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:57.572331 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:57.572336 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:57.572340 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:57.572663 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:57.573394 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:10:57.573406 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:10:57.573976 (+ 570us) client_negotiation.cc:769] Sending connection context
0620 23:10:57.574383 (+ 407us) client_negotiation.cc:240] Negotiation successful
0620 23:10:57.575128 (+ 745us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":368,"threads_started":1}
I20260620 23:10:57.598443 12842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:57.585057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:60869 (local address 127.10.191.254:36523)
0620 23:10:57.585285 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:57.585293 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:57.585903 (+ 610us) server_negotiation.cc:407] Connection header received
0620 23:10:57.586037 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:57.586045 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:57.586149 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:57.586282 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:57.587287 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:57.588295 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:57.589601 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:57.590003 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:57.590143 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:57.592360 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:57.592366 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:57.592377 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:57.592416 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:57.595890 (+ 3474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:57.596600 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:57.596607 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:57.596611 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:57.596751 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:57.597098 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:57.597105 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:57.597108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:57.597362 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:57.597508 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:57.597884 (+ 376us) server_negotiation.cc:299] Negotiation successful
0620 23:10:57.598091 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 23:10:57.598578 12921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:57.585148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:36523 (local address 127.10.191.193:60869)
0620 23:10:57.585791 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:57.585810 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:57.585938 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:57.586301 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:57.586304 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:57.586745 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:10:57.587137 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:57.587148 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:57.588427 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:57.588435 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:57.589447 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:57.589457 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:57.590187 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:57.590203 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:57.590368 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:57.591007 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:10:57.591025 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:10:57.592166 (+ 1141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:10:57.596054 (+ 3888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:57.596057 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:57.596060 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:57.596489 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:57.596868 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:57.596873 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:57.596877 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:57.596997 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:57.597473 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:10:57.597478 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:10:57.597652 (+ 174us) client_negotiation.cc:769] Sending connection context
0620 23:10:57.597793 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 23:10:57.598108 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":344,"threads_started":1}
I20260620 23:10:57.607045 12818 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.193:40525
I20260620 23:10:57.607092 12953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.193:40525 every 8 connection(s)
I20260620 23:10:57.608242 12818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
I20260620 23:10:57.610379 12954 heartbeater.cc:347] Connected to a master server at 127.10.191.254:36523
I20260620 23:10:57.610606 12954 heartbeater.cc:464] Registering TS with master...
I20260620 23:10:57.610808 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 12818
I20260620 23:10:57.611040 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance
I20260620 23:10:57.611361 12954 heartbeater.cc:511] Master 127.10.191.254:36523 requested a full tablet report, sending...
I20260620 23:10:57.611697 11007 external_mini_cluster.cc:1428] Setting key 7a15cc1dda4ff22f83533bb2244eb6a7
I20260620 23:10:57.613379 12747 ts_manager.cc:194] Registered new tserver with Master: a4532051e6c94424963f3862691fb2a4 (127.10.191.193:40525)
I20260620 23:10:57.615329 12747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.193@KRBTEST.COM'} at 127.10.191.193:49455
WARNING: no policy specified for kudu/127.10.191.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab.
Entry for principal kudu/127.10.191.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab.
I20260620 23:10:57.650077 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.194:0
--local_ip_for_outbound_sockets=127.10.191.194
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:36523
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:44933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:10:57.963068 12958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:10:57.963341 12958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:10:57.963441 12958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:10:57.972828 12958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:10:57.972963 12958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:10:57.973089 12958 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 23:10:57.973167 12958 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:10:57.973290 12958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.194
I20260620 23:10:57.988524 12958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:44933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.191.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.191.254:36523
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.194
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:10:57.990028 12958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:10:57.991411 12958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:10:58.004221 12963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:58.004334 12964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:58.004354 12966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:10:58.005169 12958 server_base.cc:1034] running on GCE node
I20260620 23:10:58.005759 12958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:10:58.006699 12958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:10:58.007921 12958 hybrid_clock.cc:648] HybridClock initialized: now 1781997058007875 us; error 62 us; skew 500 ppm
I20260620 23:10:58.008325 12958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:10:58 dist-test-slave-jkbq krb5kdc[12702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997058, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:10:58.011480 12958 init.cc:377] Logged in from keytab as kudu/127.10.191.194@KRBTEST.COM (short username kudu)
I20260620 23:10:58.012953 12958 webserver.cc:460] Webserver started at http://127.10.191.194:41185/ using document root <none> and password file <none>
I20260620 23:10:58.013619 12958 fs_manager.cc:359] Metadata directory not provided
I20260620 23:10:58.013782 12958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:10:58.014091 12958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:10:58.016993 12958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-1/data/instance:
uuid: "528ab34d28ba41f3b392a149fb596254"
format_stamp: "Formatted at 2026-06-20 23:10:58 on dist-test-slave-jkbq"
server_key: "b5ea90a817a90323223c7bfc451cf9a3"
server_key_iv: "556b30ab7cf64d8a96758b82f6055648"
server_key_version: "encryptionkey@0"
I20260620 23:10:58.017782 12958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance:
uuid: "528ab34d28ba41f3b392a149fb596254"
format_stamp: "Formatted at 2026-06-20 23:10:58 on dist-test-slave-jkbq"
server_key: "b5ea90a817a90323223c7bfc451cf9a3"
server_key_iv: "556b30ab7cf64d8a96758b82f6055648"
server_key_version: "encryptionkey@0"
I20260620 23:10:58.023029 12958 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 23:10:58.026278 12973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:58.027607 12958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:10:58.027782 12958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-1/wal
uuid: "528ab34d28ba41f3b392a149fb596254"
format_stamp: "Formatted at 2026-06-20 23:10:58 on dist-test-slave-jkbq"
server_key: "b5ea90a817a90323223c7bfc451cf9a3"
server_key_iv: "556b30ab7cf64d8a96758b82f6055648"
server_key_version: "encryptionkey@0"
I20260620 23:10:58.027977 12958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:10:58.046555 12958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:10:58.050463 12958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:10:58.050758 12958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:10:58.053329 12958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:10:58.053479 12958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:58.053620 12958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:10:58.053721 12958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:10:58 dist-test-slave-jkbq krb5kdc[12702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997058, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:10:58.077813 12842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:58.054141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:39579 (local address 127.10.191.254:36523)
0620 23:10:58.054309 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:58.054315 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:58.055851 (+ 1536us) server_negotiation.cc:407] Connection header received
0620 23:10:58.059143 (+ 3292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:58.059150 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:58.059262 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:58.059384 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:58.061097 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:58.062053 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:58.064382 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:58.064998 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:58.065138 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:58.070313 (+ 5175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:58.070319 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:58.070324 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:58.070372 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:58.073847 (+ 3475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:58.074946 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:58.074953 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:58.074959 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:58.075113 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:58.075770 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:58.075779 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:58.075781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:58.076045 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:58.076219 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:58.077236 (+ 1017us) server_negotiation.cc:299] Negotiation successful
0620 23:10:58.077460 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 23:10:58.079365 12982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:58.054706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:36523 (local address 127.10.191.194:39579)
0620 23:10:58.055670 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:58.055727 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:58.058714 (+ 2987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:58.059675 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:58.059683 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:58.060279 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:10:58.060914 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:58.060931 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:58.062305 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:58.062312 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:58.064232 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:58.064244 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:58.065241 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:58.065250 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:58.065489 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:58.066532 (+ 1043us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:10:58.066555 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:10:58.070000 (+ 3445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:10:58.073999 (+ 3999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:58.074008 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:58.074029 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:58.074713 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:58.075224 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:58.075229 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:58.075233 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:58.075608 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:58.076276 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:10:58.076285 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:10:58.076742 (+ 457us) client_negotiation.cc:769] Sending connection context
0620 23:10:58.077127 (+ 385us) client_negotiation.cc:240] Negotiation successful
0620 23:10:58.077723 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":425,"threads_started":1}
I20260620 23:10:58.103534 12842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:58.089539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:50139 (local address 127.10.191.254:36523)
0620 23:10:58.089736 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:58.089744 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:58.090483 (+ 739us) server_negotiation.cc:407] Connection header received
0620 23:10:58.090722 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:58.090730 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:58.090822 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:58.090961 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:58.092521 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:58.093558 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:58.095752 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:58.096222 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:58.096375 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:58.098238 (+ 1863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:58.098243 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:58.098246 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:58.098274 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:58.100779 (+ 2505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:58.101429 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:58.101440 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:58.101442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:58.101558 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:58.101940 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:58.101946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:58.101949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:58.102414 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:58.102564 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:58.102985 (+ 421us) server_negotiation.cc:299] Negotiation successful
0620 23:10:58.103188 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 23:10:58.103768 13069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:58.089629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:36523 (local address 127.10.191.194:50139)
0620 23:10:58.090373 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:58.090395 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:58.090557 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:58.091152 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:58.091156 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:58.091821 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:10:58.092317 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:58.092333 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:58.093711 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:58.093719 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:58.095606 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:58.095618 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:58.096445 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:58.096452 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:58.096621 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:58.097351 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:10:58.097367 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:10:58.098050 (+ 683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:10:58.100914 (+ 2864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:58.100919 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:58.100923 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:58.101303 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:58.101667 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:58.101673 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:58.101677 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:58.101810 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:58.102526 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:10:58.102531 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:10:58.102723 (+ 192us) client_negotiation.cc:769] Sending connection context
0620 23:10:58.102870 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 23:10:58.103253 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":467,"threads_started":1}
I20260620 23:10:58.104622 12958 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.194:45137
I20260620 23:10:58.104702 13092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.194:45137 every 8 connection(s)
I20260620 23:10:58.105965 12958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
I20260620 23:10:58.107052 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 12958
I20260620 23:10:58.107228 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance
I20260620 23:10:58.107753 11007 external_mini_cluster.cc:1428] Setting key 9fc0ba823d832909081651d66f36d389
I20260620 23:10:58.108628 13093 heartbeater.cc:347] Connected to a master server at 127.10.191.254:36523
I20260620 23:10:58.109360 13093 heartbeater.cc:464] Registering TS with master...
I20260620 23:10:58.110044 13093 heartbeater.cc:511] Master 127.10.191.254:36523 requested a full tablet report, sending...
I20260620 23:10:58.111336 12747 ts_manager.cc:194] Registered new tserver with Master: 528ab34d28ba41f3b392a149fb596254 (127.10.191.194:45137)
I20260620 23:10:58.112102 12747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.194@KRBTEST.COM'} at 127.10.191.194:39579
WARNING: no policy specified for kudu/127.10.191.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab.
Entry for principal kudu/127.10.191.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab.
I20260620 23:10:58.146580 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.195:0
--local_ip_for_outbound_sockets=127.10.191.195
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:36523
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:44933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:10:58.467259 13097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:10:58.467509 13097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:10:58.467711 13097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:10:58.474712 13097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:10:58.474843 13097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:10:58.475023 13097 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 23:10:58.475107 13097 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:10:58.475239 13097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.195
I20260620 23:10:58.485044 13097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:44933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.191.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.191.254:36523
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.195
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:10:58.486251 13097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:10:58.487617 13097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:10:58.500608 13103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:58.500581 13105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:10:58.500659 13102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:10:58.501349 13097 server_base.cc:1034] running on GCE node
I20260620 23:10:58.501972 13097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:10:58.503022 13097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:10:58.504251 13097 hybrid_clock.cc:648] HybridClock initialized: now 1781997058504206 us; error 52 us; skew 500 ppm
I20260620 23:10:58.504568 13097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:10:58 dist-test-slave-jkbq krb5kdc[12702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997058, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:10:58.507663 13097 init.cc:377] Logged in from keytab as kudu/127.10.191.195@KRBTEST.COM (short username kudu)
I20260620 23:10:58.509162 13097 webserver.cc:460] Webserver started at http://127.10.191.195:45685/ using document root <none> and password file <none>
I20260620 23:10:58.509917 13097 fs_manager.cc:359] Metadata directory not provided
I20260620 23:10:58.510059 13097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:10:58.510370 13097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:10:58.513343 13097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-2/data/instance:
uuid: "5285a4ac2d32487890c7b633e69088fe"
format_stamp: "Formatted at 2026-06-20 23:10:58 on dist-test-slave-jkbq"
server_key: "d5cfbd4a282303876d0690816bf24fa0"
server_key_iv: "c796aebfc1c11db8c0bd9ab7ca0fcca4"
server_key_version: "encryptionkey@0"
I20260620 23:10:58.514075 13097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance:
uuid: "5285a4ac2d32487890c7b633e69088fe"
format_stamp: "Formatted at 2026-06-20 23:10:58 on dist-test-slave-jkbq"
server_key: "d5cfbd4a282303876d0690816bf24fa0"
server_key_iv: "c796aebfc1c11db8c0bd9ab7ca0fcca4"
server_key_version: "encryptionkey@0"
I20260620 23:10:58.519177 13097 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 23:10:58.522732 13112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:58.524121 13097 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:10:58.524303 13097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-2/wal
uuid: "5285a4ac2d32487890c7b633e69088fe"
format_stamp: "Formatted at 2026-06-20 23:10:58 on dist-test-slave-jkbq"
server_key: "d5cfbd4a282303876d0690816bf24fa0"
server_key_iv: "c796aebfc1c11db8c0bd9ab7ca0fcca4"
server_key_version: "encryptionkey@0"
I20260620 23:10:58.524488 13097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:10:58.535892 13097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:10:58.539887 13097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:10:58.540176 13097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:10:58.542645 13097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:10:58.542804 13097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:10:58.542989 13097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:10:58.543134 13097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:10:58 dist-test-slave-jkbq krb5kdc[12702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997058, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:10:58.567083 12842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:58.543492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:53655 (local address 127.10.191.254:36523)
0620 23:10:58.543686 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:58.543692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:58.545168 (+ 1476us) server_negotiation.cc:407] Connection header received
0620 23:10:58.547168 (+ 2000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:58.547174 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:58.547262 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:58.547393 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:58.550069 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:58.550930 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:58.554290 (+ 3360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:58.554794 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:58.554991 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:58.559930 (+ 4939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:58.559935 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:58.559939 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:58.559970 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:58.563390 (+ 3420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:58.564380 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:58.564386 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:58.564390 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:58.564594 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:58.565176 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:58.565182 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:58.565185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:58.565468 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:58.565617 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:58.566459 (+ 842us) server_negotiation.cc:299] Negotiation successful
0620 23:10:58.566690 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 23:10:58.568620 13121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:58.544180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:36523 (local address 127.10.191.195:53655)
0620 23:10:58.544986 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:58.545040 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:58.546838 (+ 1798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:58.547704 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:58.547720 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:58.548660 (+ 940us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:10:58.549857 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:58.549890 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:58.551133 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:58.551142 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:58.554036 (+ 2894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:58.554055 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:58.555112 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:58.555123 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:58.555514 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:58.556326 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:10:58.556367 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:10:58.559672 (+ 3305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:10:58.563546 (+ 3874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:58.563559 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:58.563576 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:58.564236 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:58.564762 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:58.564767 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:58.564771 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:58.565039 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:58.565612 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:10:58.565620 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:10:58.566086 (+ 466us) client_negotiation.cc:769] Sending connection context
0620 23:10:58.566359 (+ 273us) client_negotiation.cc:240] Negotiation successful
0620 23:10:58.566955 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":366,"threads_started":1}
I20260620 23:10:58.589236 12842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:58.574084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:48007 (local address 127.10.191.254:36523)
0620 23:10:58.574234 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:58.574238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:58.574989 (+ 751us) server_negotiation.cc:407] Connection header received
0620 23:10:58.575203 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:58.575211 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:58.575314 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:58.575446 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:58.576333 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:58.577789 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:58.579151 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:58.579837 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:58.579982 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:58.582657 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:58.582663 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:58.582668 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:58.582718 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:58.586234 (+ 3516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:58.587218 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:58.587226 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:58.587231 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:58.587411 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:58.587847 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:58.587852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:58.587856 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:58.588109 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:58.588257 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:58.588663 (+ 406us) server_negotiation.cc:299] Negotiation successful
0620 23:10:58.588881 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 23:10:58.589362 13181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:58.574232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:36523 (local address 127.10.191.195:48007)
0620 23:10:58.574829 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:58.574847 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:58.575045 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:58.575481 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:58.575486 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:58.575924 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:10:58.576191 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:58.576200 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:58.577942 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:58.577946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:58.579001 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:58.579012 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:58.580084 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:58.580096 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:58.580407 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:58.581223 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:10:58.581251 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:10:58.582466 (+ 1215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:10:58.586362 (+ 3896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:58.586371 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:58.586378 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:58.587088 (+ 710us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:58.587579 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:58.587584 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:58.587588 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:58.587726 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:58.588230 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:10:58.588235 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:10:58.588423 (+ 188us) client_negotiation.cc:769] Sending connection context
0620 23:10:58.588563 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 23:10:58.588917 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":346,"threads_started":1}
I20260620 23:10:58.599541 13097 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.195:42307
I20260620 23:10:58.599560 13231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.195:42307 every 8 connection(s)
I20260620 23:10:58.600733 13097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
I20260620 23:10:58.602566 13232 heartbeater.cc:347] Connected to a master server at 127.10.191.254:36523
I20260620 23:10:58.602798 13232 heartbeater.cc:464] Registering TS with master...
I20260620 23:10:58.603434 13232 heartbeater.cc:511] Master 127.10.191.254:36523 requested a full tablet report, sending...
I20260620 23:10:58.604427 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 13097
I20260620 23:10:58.604602 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance
I20260620 23:10:58.604852 12747 ts_manager.cc:194] Registered new tserver with Master: 5285a4ac2d32487890c7b633e69088fe (127.10.191.195:42307)
I20260620 23:10:58.605104 11007 external_mini_cluster.cc:1428] Setting key ffe59760020929ad472cbaab41d8658a
I20260620 23:10:58.605708 12747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.195@KRBTEST.COM'} at 127.10.191.195:53655
I20260620 23:10:58.608227 11007 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:10:58.618719 12954 heartbeater.cc:503] Master 127.10.191.254:36523 was elected leader, sending a full tablet report...
Jun 20 23:10:58 dist-test-slave-jkbq krb5kdc[12702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 23:10:58 dist-test-slave-jkbq krb5kdc[12702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:10:58.652803 12842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:58.640063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58166 (local address 127.10.191.254:36523)
0620 23:10:58.640233 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:58.640237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:58.640635 (+ 398us) server_negotiation.cc:407] Connection header received
0620 23:10:58.640760 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:58.640764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:58.640835 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:58.640922 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:58.642224 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:58.643028 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:58.644265 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:58.644686 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:58.644780 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:58.648360 (+ 3580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:58.648364 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:58.648368 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:58.648398 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:58.650665 (+ 2267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:58.651323 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:58.651326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:58.651328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:58.651446 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:58.651749 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:58.651752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:58.651754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:58.652005 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:58.652180 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:58.652394 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 23:10:58.652549 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 23:10:58.656612 12746 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58166:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:10:58.663478 12746 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:10:58.685678 13248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:58.676433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45896 (local address 127.10.191.193:40525)
0620 23:10:58.677089 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:58.677098 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:58.677798 (+ 700us) server_negotiation.cc:407] Connection header received
0620 23:10:58.677882 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:58.677888 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:58.678108 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:58.678275 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:10:58.679288 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:58.680973 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:58.683362 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:58.684595 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:58.684786 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:58.684977 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:58.685211 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 23:10:58.685367 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":381,"threads_started":1}
I20260620 23:10:58.685811 13245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:58.675906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:40525 (local address 127.0.0.1:45896)
0620 23:10:58.677442 (+ 1536us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:58.677457 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:58.677620 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:58.678518 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:58.678528 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:58.678563 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:10:58.679054 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:58.679073 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:58.681109 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:58.681114 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:58.683134 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:58.683146 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:58.684797 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:58.684803 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:58.684972 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:58.685024 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 23:10:58.685177 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 23:10:58.685367 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1364,"mutex_wait_us":3,"spinlock_wait_cycles":4224,"thread_start_us":1069,"threads_started":1}
I20260620 23:10:58.686431 13249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:58.677373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.194:45137 (local address 127.0.0.1:51198)
0620 23:10:58.678068 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:58.678085 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:58.678211 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:58.679275 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:58.679280 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:58.679312 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:10:58.679733 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:58.679745 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:58.681468 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:58.681475 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:58.683844 (+ 2369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:58.683858 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:58.685607 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:58.685613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:58.685804 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:58.685838 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 23:10:58.685907 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 23:10:58.686047 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"mutex_wait_us":26,"thread_start_us":629,"threads_started":1}
I20260620 23:10:58.686792 13246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:58.675530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43302 (local address 127.10.191.195:42307)
0620 23:10:58.676671 (+ 1141us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:58.676684 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:58.677453 (+ 769us) server_negotiation.cc:407] Connection header received
0620 23:10:58.677856 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:58.677865 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:58.678106 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:58.678268 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:10:58.679315 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:58.680959 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:58.684012 (+ 3053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:58.685396 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:58.685573 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:58.685740 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:58.686294 (+ 554us) server_negotiation.cc:299] Negotiation successful
0620 23:10:58.686479 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":955,"thread_start_us":793,"threads_started":1}
I20260620 23:10:58.687492 13250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:58.677837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51198 (local address 127.10.191.194:45137)
0620 23:10:58.678727 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:58.678737 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:58.678764 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:10:58.678858 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:58.678910 (+ 52us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:58.679087 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:58.679290 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:10:58.680035 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:58.681326 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:58.684033 (+ 2707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:58.685370 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:58.686793 (+ 1423us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:58.686997 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:58.687060 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 23:10:58.687203 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":725,"thread_start_us":603,"threads_started":1}
I20260620 23:10:58.687557 13247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:58.675314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.195:42307 (local address 127.0.0.1:43302)
0620 23:10:58.677232 (+ 1918us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:58.677306 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:58.677620 (+ 314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:58.678582 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:58.678586 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:58.678611 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:10:58.679079 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:58.679089 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:58.681174 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:58.681178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:58.683845 (+ 2667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:58.683858 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:58.685926 (+ 2068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:58.685932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:58.686097 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:58.686128 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 23:10:58.687060 (+ 932us) client_negotiation.cc:240] Negotiation successful
0620 23:10:58.687188 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1545,"mutex_wait_us":144,"spinlock_wait_cycles":9472,"thread_start_us":787,"threads_started":1}
I20260620 23:10:58.689894 12884 tablet_service.cc:1467] Processing CreateTablet for tablet 956bab8764154b55abc7a626260480d4 (DEFAULT_TABLE table=test-table [id=3496e3835b484dd1ae2391912395dc1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:10:58.690275 13023 tablet_service.cc:1467] Processing CreateTablet for tablet 956bab8764154b55abc7a626260480d4 (DEFAULT_TABLE table=test-table [id=3496e3835b484dd1ae2391912395dc1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:10:58.691470 12884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 956bab8764154b55abc7a626260480d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:58.690999 13162 tablet_service.cc:1467] Processing CreateTablet for tablet 956bab8764154b55abc7a626260480d4 (DEFAULT_TABLE table=test-table [id=3496e3835b484dd1ae2391912395dc1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:10:58.691584 13023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 956bab8764154b55abc7a626260480d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:58.692251 13162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 956bab8764154b55abc7a626260480d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:58.703574 13251 tablet_bootstrap.cc:492] T 956bab8764154b55abc7a626260480d4 P 528ab34d28ba41f3b392a149fb596254: Bootstrap starting.
I20260620 23:10:58.705791 13252 tablet_bootstrap.cc:492] T 956bab8764154b55abc7a626260480d4 P a4532051e6c94424963f3862691fb2a4: Bootstrap starting.
I20260620 23:10:58.706580 13253 tablet_bootstrap.cc:492] T 956bab8764154b55abc7a626260480d4 P 5285a4ac2d32487890c7b633e69088fe: Bootstrap starting.
I20260620 23:10:58.707294 13251 tablet_bootstrap.cc:654] T 956bab8764154b55abc7a626260480d4 P 528ab34d28ba41f3b392a149fb596254: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:58.708611 13251 log.cc:824] T 956bab8764154b55abc7a626260480d4 P 528ab34d28ba41f3b392a149fb596254: Log is configured to *not* fsync() on all Append() calls
I20260620 23:10:58.710067 13252 tablet_bootstrap.cc:654] T 956bab8764154b55abc7a626260480d4 P a4532051e6c94424963f3862691fb2a4: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:58.710498 13253 tablet_bootstrap.cc:654] T 956bab8764154b55abc7a626260480d4 P 5285a4ac2d32487890c7b633e69088fe: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:58.711277 13252 log.cc:824] T 956bab8764154b55abc7a626260480d4 P a4532051e6c94424963f3862691fb2a4: Log is configured to *not* fsync() on all Append() calls
I20260620 23:10:58.711635 13253 log.cc:824] T 956bab8764154b55abc7a626260480d4 P 5285a4ac2d32487890c7b633e69088fe: Log is configured to *not* fsync() on all Append() calls
I20260620 23:10:58.713624 13251 tablet_bootstrap.cc:492] T 956bab8764154b55abc7a626260480d4 P 528ab34d28ba41f3b392a149fb596254: No bootstrap required, opened a new log
I20260620 23:10:58.713903 13251 ts_tablet_manager.cc:1397] T 956bab8764154b55abc7a626260480d4 P 528ab34d28ba41f3b392a149fb596254: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.002s
I20260620 23:10:58.714141 13253 tablet_bootstrap.cc:492] T 956bab8764154b55abc7a626260480d4 P 5285a4ac2d32487890c7b633e69088fe: No bootstrap required, opened a new log
I20260620 23:10:58.714216 13252 tablet_bootstrap.cc:492] T 956bab8764154b55abc7a626260480d4 P a4532051e6c94424963f3862691fb2a4: No bootstrap required, opened a new log
I20260620 23:10:58.714385 13253 ts_tablet_manager.cc:1397] T 956bab8764154b55abc7a626260480d4 P 5285a4ac2d32487890c7b633e69088fe: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260620 23:10:58.714483 13252 ts_tablet_manager.cc:1397] T 956bab8764154b55abc7a626260480d4 P a4532051e6c94424963f3862691fb2a4: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260620 23:10:58.721903 13253 raft_consensus.cc:348] T 956bab8764154b55abc7a626260480d4 P 5285a4ac2d32487890c7b633e69088fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } } peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } }
I20260620 23:10:58.722345 13253 raft_consensus.cc:374] T 956bab8764154b55abc7a626260480d4 P 5285a4ac2d32487890c7b633e69088fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:10:58.722476 13253 raft_consensus.cc:729] T 956bab8764154b55abc7a626260480d4 P 5285a4ac2d32487890c7b633e69088fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5285a4ac2d32487890c7b633e69088fe, State: Initialized, Role: FOLLOWER
I20260620 23:10:58.723156 13253 consensus_queue.cc:260] T 956bab8764154b55abc7a626260480d4 P 5285a4ac2d32487890c7b633e69088fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } } peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } }
I20260620 23:10:58.724228 13232 heartbeater.cc:503] Master 127.10.191.254:36523 was elected leader, sending a full tablet report...
I20260620 23:10:58.724383 13251 raft_consensus.cc:348] T 956bab8764154b55abc7a626260480d4 P 528ab34d28ba41f3b392a149fb596254 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } } peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } }
I20260620 23:10:58.724785 13251 raft_consensus.cc:374] T 956bab8764154b55abc7a626260480d4 P 528ab34d28ba41f3b392a149fb596254 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:10:58.724879 13251 raft_consensus.cc:729] T 956bab8764154b55abc7a626260480d4 P 528ab34d28ba41f3b392a149fb596254 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 528ab34d28ba41f3b392a149fb596254, State: Initialized, Role: FOLLOWER
I20260620 23:10:58.724901 13253 ts_tablet_manager.cc:1428] T 956bab8764154b55abc7a626260480d4 P 5285a4ac2d32487890c7b633e69088fe: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260620 23:10:58.724997 13252 raft_consensus.cc:348] T 956bab8764154b55abc7a626260480d4 P a4532051e6c94424963f3862691fb2a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } } peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } }
I20260620 23:10:58.725487 13252 raft_consensus.cc:374] T 956bab8764154b55abc7a626260480d4 P a4532051e6c94424963f3862691fb2a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:10:58.725589 13252 raft_consensus.cc:729] T 956bab8764154b55abc7a626260480d4 P a4532051e6c94424963f3862691fb2a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4532051e6c94424963f3862691fb2a4, State: Initialized, Role: FOLLOWER
I20260620 23:10:58.725466 13251 consensus_queue.cc:260] T 956bab8764154b55abc7a626260480d4 P 528ab34d28ba41f3b392a149fb596254 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } } peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } }
I20260620 23:10:58.726200 13252 consensus_queue.cc:260] T 956bab8764154b55abc7a626260480d4 P a4532051e6c94424963f3862691fb2a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } } peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } }
I20260620 23:10:58.728322 13252 ts_tablet_manager.cc:1428] T 956bab8764154b55abc7a626260480d4 P a4532051e6c94424963f3862691fb2a4: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260620 23:10:58.729979 13093 heartbeater.cc:503] Master 127.10.191.254:36523 was elected leader, sending a full tablet report...
I20260620 23:10:58.732012 13251 ts_tablet_manager.cc:1428] T 956bab8764154b55abc7a626260480d4 P 528ab34d28ba41f3b392a149fb596254: Time spent starting tablet: real 0.018s user 0.016s sys 0.002s
I20260620 23:10:58.841861 13259 raft_consensus.cc:482] T 956bab8764154b55abc7a626260480d4 P a4532051e6c94424963f3862691fb2a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:10:58.842176 13259 raft_consensus.cc:504] T 956bab8764154b55abc7a626260480d4 P a4532051e6c94424963f3862691fb2a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } } peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } }
I20260620 23:10:58.843782 13259 leader_election.cc:288] T 956bab8764154b55abc7a626260480d4 P a4532051e6c94424963f3862691fb2a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5285a4ac2d32487890c7b633e69088fe (127.10.191.195:42307), 528ab34d28ba41f3b392a149fb596254 (127.10.191.194:45137)
W20260620 23:10:58.852267 13233 tablet.cc:2369] T 956bab8764154b55abc7a626260480d4 P 5285a4ac2d32487890c7b633e69088fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:10:58.852952 13250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:58.844308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:34303 (local address 127.10.191.194:45137)
0620 23:10:58.844511 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:58.844517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:58.845267 (+ 750us) server_negotiation.cc:407] Connection header received
0620 23:10:58.845460 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:58.845466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:58.845545 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:58.845649 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:10:58.846491 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:58.847611 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:58.850649 (+ 3038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:58.851903 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:58.852062 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:58.852135 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:58.852549 (+ 414us) server_negotiation.cc:299] Negotiation successful
0620 23:10:58.852688 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:10:58.853171 13261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:58.844253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.195:42307 (local address 127.10.191.193:51345)
0620 23:10:58.845127 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:58.845142 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:58.845331 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:58.845682 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:58.845687 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:58.845734 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:10:58.846302 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:58.846319 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:58.847742 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:58.847748 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:58.850456 (+ 2708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:58.850471 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:58.852305 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:58.852309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:58.852478 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:58.852510 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 23:10:58.852629 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 23:10:58.852758 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"mutex_wait_us":178,"spinlock_wait_cycles":47232,"thread_start_us":489,"threads_started":1}
I20260620 23:10:58.853257 13246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:58.844309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:51345 (local address 127.10.191.195:42307)
0620 23:10:58.844511 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:58.844517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:58.845267 (+ 750us) server_negotiation.cc:407] Connection header received
0620 23:10:58.845460 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:58.845466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:58.845544 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:58.845649 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:10:58.846487 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:58.847607 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:58.850649 (+ 3042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:58.851891 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:58.852063 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:58.852135 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:58.852654 (+ 519us) server_negotiation.cc:299] Negotiation successful
0620 23:10:58.852772 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 23:10:58.853683 13260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:58.844138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.194:45137 (local address 127.10.191.193:34303)
0620 23:10:58.845106 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:58.845131 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:58.845295 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:58.845682 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:58.845687 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:58.845733 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:10:58.846336 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:58.846351 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:58.847744 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:58.847748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:58.850462 (+ 2714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:58.850471 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:58.852115 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:58.852121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:58.852329 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:58.852366 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 23:10:58.853206 (+ 840us) client_negotiation.cc:240] Negotiation successful
0620 23:10:58.853379 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"mutex_wait_us":239,"spinlock_wait_cycles":17024,"thread_start_us":584,"threads_started":1}
I20260620 23:10:58.854133 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "956bab8764154b55abc7a626260480d4" candidate_uuid: "a4532051e6c94424963f3862691fb2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5285a4ac2d32487890c7b633e69088fe" is_pre_election: true
I20260620 23:10:58.854624 13187 raft_consensus.cc:2454] T 956bab8764154b55abc7a626260480d4 P 5285a4ac2d32487890c7b633e69088fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4532051e6c94424963f3862691fb2a4 in term 0.
I20260620 23:10:58.854604 13043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "956bab8764154b55abc7a626260480d4" candidate_uuid: "a4532051e6c94424963f3862691fb2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "528ab34d28ba41f3b392a149fb596254" is_pre_election: true
I20260620 23:10:58.855110 13043 raft_consensus.cc:2454] T 956bab8764154b55abc7a626260480d4 P 528ab34d28ba41f3b392a149fb596254 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4532051e6c94424963f3862691fb2a4 in term 0.
I20260620 23:10:58.855358 12836 leader_election.cc:302] T 956bab8764154b55abc7a626260480d4 P a4532051e6c94424963f3862691fb2a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5285a4ac2d32487890c7b633e69088fe, a4532051e6c94424963f3862691fb2a4; no voters:
I20260620 23:10:58.855737 13259 raft_consensus.cc:2781] T 956bab8764154b55abc7a626260480d4 P a4532051e6c94424963f3862691fb2a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:10:58.855880 13259 raft_consensus.cc:482] T 956bab8764154b55abc7a626260480d4 P a4532051e6c94424963f3862691fb2a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:10:58.856009 13259 raft_consensus.cc:3037] T 956bab8764154b55abc7a626260480d4 P a4532051e6c94424963f3862691fb2a4 [term 0 FOLLOWER]: Advancing to term 1
W20260620 23:10:58.858152 13094 tablet.cc:2369] T 956bab8764154b55abc7a626260480d4 P 528ab34d28ba41f3b392a149fb596254: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:10:58.858319 13259 raft_consensus.cc:504] T 956bab8764154b55abc7a626260480d4 P a4532051e6c94424963f3862691fb2a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } } peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } }
I20260620 23:10:58.859139 13259 leader_election.cc:288] T 956bab8764154b55abc7a626260480d4 P a4532051e6c94424963f3862691fb2a4 [CANDIDATE]: Term 1 election: Requested vote from peers 5285a4ac2d32487890c7b633e69088fe (127.10.191.195:42307), 528ab34d28ba41f3b392a149fb596254 (127.10.191.194:45137)
I20260620 23:10:58.859485 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "956bab8764154b55abc7a626260480d4" candidate_uuid: "a4532051e6c94424963f3862691fb2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5285a4ac2d32487890c7b633e69088fe"
I20260620 23:10:58.859587 13043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "956bab8764154b55abc7a626260480d4" candidate_uuid: "a4532051e6c94424963f3862691fb2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "528ab34d28ba41f3b392a149fb596254"
I20260620 23:10:58.859689 13187 raft_consensus.cc:3037] T 956bab8764154b55abc7a626260480d4 P 5285a4ac2d32487890c7b633e69088fe [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:10:58.859776 13043 raft_consensus.cc:3037] T 956bab8764154b55abc7a626260480d4 P 528ab34d28ba41f3b392a149fb596254 [term 0 FOLLOWER]: Advancing to term 1
W20260620 23:10:58.860530 12955 tablet.cc:2369] T 956bab8764154b55abc7a626260480d4 P a4532051e6c94424963f3862691fb2a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:10:58.862911 13043 raft_consensus.cc:2454] T 956bab8764154b55abc7a626260480d4 P 528ab34d28ba41f3b392a149fb596254 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4532051e6c94424963f3862691fb2a4 in term 1.
I20260620 23:10:58.862911 13187 raft_consensus.cc:2454] T 956bab8764154b55abc7a626260480d4 P 5285a4ac2d32487890c7b633e69088fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4532051e6c94424963f3862691fb2a4 in term 1.
I20260620 23:10:58.863397 12835 leader_election.cc:302] T 956bab8764154b55abc7a626260480d4 P a4532051e6c94424963f3862691fb2a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 528ab34d28ba41f3b392a149fb596254, a4532051e6c94424963f3862691fb2a4; no voters:
I20260620 23:10:58.863734 13259 raft_consensus.cc:2781] T 956bab8764154b55abc7a626260480d4 P a4532051e6c94424963f3862691fb2a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:10:58.864166 13259 raft_consensus.cc:686] T 956bab8764154b55abc7a626260480d4 P a4532051e6c94424963f3862691fb2a4 [term 1 LEADER]: Becoming Leader. State: Replica: a4532051e6c94424963f3862691fb2a4, State: Running, Role: LEADER
I20260620 23:10:58.864636 13259 consensus_queue.cc:237] T 956bab8764154b55abc7a626260480d4 P a4532051e6c94424963f3862691fb2a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } } peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } }
I20260620 23:10:58.869445 12746 catalog_manager.cc:5495] T 956bab8764154b55abc7a626260480d4 P a4532051e6c94424963f3862691fb2a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4532051e6c94424963f3862691fb2a4 (127.10.191.193). New cstate: current_term: 1 leader_uuid: "a4532051e6c94424963f3862691fb2a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } health_report { overall_health: UNKNOWN } } }
Jun 20 23:10:58 dist-test-slave-jkbq krb5kdc[12702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997056, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:10:58.963089 12842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:58.949445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58182 (local address 127.10.191.254:36523)
0620 23:10:58.949772 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:58.949781 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:58.950206 (+ 425us) server_negotiation.cc:407] Connection header received
0620 23:10:58.950418 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:58.950421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:58.950486 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:58.950588 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:10:58.951892 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:58.952615 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:58.953890 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:58.954288 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:58.954434 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:58.957665 (+ 3231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:10:58.957670 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:10:58.957673 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:10:58.957699 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:10:58.960386 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:58.961070 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:58.961078 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:58.961083 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:58.961209 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:10:58.961558 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:10:58.961566 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:10:58.961569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:10:58.961844 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:10:58.962000 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:58.962403 (+ 403us) server_negotiation.cc:299] Negotiation successful
0620 23:10:58.962742 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":141,"server-negotiator.queue_time_us":143}
I20260620 23:10:58.963122 13269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:58.949456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:36523 (local address 127.0.0.1:58182)
0620 23:10:58.950098 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:58.950113 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:58.950285 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:58.950609 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:58.950613 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:58.951472 (+ 859us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:10:58.951732 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:58.951742 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:58.952748 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:58.952756 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:58.953721 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:58.953733 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:58.954474 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:58.954480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:58.954656 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:58.955386 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:10:58.955403 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:10:58.957481 (+ 2078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:10:58.960541 (+ 3060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:58.960554 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:58.960563 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:58.960955 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:58.961315 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:10:58.961322 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:10:58.961325 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:10:58.961457 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:10:58.961974 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:10:58.961983 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:10:58.962149 (+ 166us) client_negotiation.cc:769] Sending connection context
0620 23:10:58.962305 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 23:10:58.962609 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":382,"threads_started":1}
I20260620 23:10:58.968503 12746 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:58182:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 23:10:58.969662 12746 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:10:58.976974 12884 tablet_service.cc:1467] Processing CreateTablet for tablet c2646e3b08404b6180fc367197e395de (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fd743308a61442d4a1e4876346cb463e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:10:58.976951 13023 tablet_service.cc:1467] Processing CreateTablet for tablet c2646e3b08404b6180fc367197e395de (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fd743308a61442d4a1e4876346cb463e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:10:58.977105 13162 tablet_service.cc:1467] Processing CreateTablet for tablet c2646e3b08404b6180fc367197e395de (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fd743308a61442d4a1e4876346cb463e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:10:58.977865 12884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2646e3b08404b6180fc367197e395de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:58.977874 13162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2646e3b08404b6180fc367197e395de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:58.977875 13023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2646e3b08404b6180fc367197e395de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:58.983952 13252 tablet_bootstrap.cc:492] T c2646e3b08404b6180fc367197e395de P a4532051e6c94424963f3862691fb2a4: Bootstrap starting.
I20260620 23:10:58.984935 13251 tablet_bootstrap.cc:492] T c2646e3b08404b6180fc367197e395de P 528ab34d28ba41f3b392a149fb596254: Bootstrap starting.
I20260620 23:10:58.984982 13253 tablet_bootstrap.cc:492] T c2646e3b08404b6180fc367197e395de P 5285a4ac2d32487890c7b633e69088fe: Bootstrap starting.
I20260620 23:10:58.985579 13252 tablet_bootstrap.cc:654] T c2646e3b08404b6180fc367197e395de P a4532051e6c94424963f3862691fb2a4: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:58.986268 13251 tablet_bootstrap.cc:654] T c2646e3b08404b6180fc367197e395de P 528ab34d28ba41f3b392a149fb596254: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:58.986640 13253 tablet_bootstrap.cc:654] T c2646e3b08404b6180fc367197e395de P 5285a4ac2d32487890c7b633e69088fe: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:58.988366 13251 tablet_bootstrap.cc:492] T c2646e3b08404b6180fc367197e395de P 528ab34d28ba41f3b392a149fb596254: No bootstrap required, opened a new log
I20260620 23:10:58.988531 13251 ts_tablet_manager.cc:1397] T c2646e3b08404b6180fc367197e395de P 528ab34d28ba41f3b392a149fb596254: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 23:10:58.988606 13252 tablet_bootstrap.cc:492] T c2646e3b08404b6180fc367197e395de P a4532051e6c94424963f3862691fb2a4: No bootstrap required, opened a new log
I20260620 23:10:58.988788 13252 ts_tablet_manager.cc:1397] T c2646e3b08404b6180fc367197e395de P a4532051e6c94424963f3862691fb2a4: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260620 23:10:58.989259 13251 raft_consensus.cc:348] T c2646e3b08404b6180fc367197e395de P 528ab34d28ba41f3b392a149fb596254 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } }
I20260620 23:10:58.989478 13251 raft_consensus.cc:374] T c2646e3b08404b6180fc367197e395de P 528ab34d28ba41f3b392a149fb596254 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:10:58.989573 13251 raft_consensus.cc:729] T c2646e3b08404b6180fc367197e395de P 528ab34d28ba41f3b392a149fb596254 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 528ab34d28ba41f3b392a149fb596254, State: Initialized, Role: FOLLOWER
I20260620 23:10:58.989817 13251 consensus_queue.cc:260] T c2646e3b08404b6180fc367197e395de P 528ab34d28ba41f3b392a149fb596254 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } }
I20260620 23:10:58.989781 13252 raft_consensus.cc:348] T c2646e3b08404b6180fc367197e395de P a4532051e6c94424963f3862691fb2a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } }
I20260620 23:10:58.990079 13252 raft_consensus.cc:374] T c2646e3b08404b6180fc367197e395de P a4532051e6c94424963f3862691fb2a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:10:58.990193 13252 raft_consensus.cc:729] T c2646e3b08404b6180fc367197e395de P a4532051e6c94424963f3862691fb2a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4532051e6c94424963f3862691fb2a4, State: Initialized, Role: FOLLOWER
I20260620 23:10:58.990258 13258 tablet_replica.cc:439] T c2646e3b08404b6180fc367197e395de P 528ab34d28ba41f3b392a149fb596254: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } } }
I20260620 23:10:58.990614 13258 tablet_replica.cc:442] T c2646e3b08404b6180fc367197e395de P 528ab34d28ba41f3b392a149fb596254: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:10:58.990455 13252 consensus_queue.cc:260] T c2646e3b08404b6180fc367197e395de P a4532051e6c94424963f3862691fb2a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } }
I20260620 23:10:58.991293 13251 ts_tablet_manager.cc:1428] T c2646e3b08404b6180fc367197e395de P 528ab34d28ba41f3b392a149fb596254: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 23:10:58.991432 13259 tablet_replica.cc:439] T c2646e3b08404b6180fc367197e395de P a4532051e6c94424963f3862691fb2a4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } } }
I20260620 23:10:58.991814 13259 tablet_replica.cc:442] T c2646e3b08404b6180fc367197e395de P a4532051e6c94424963f3862691fb2a4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:10:58.992290 13252 ts_tablet_manager.cc:1428] T c2646e3b08404b6180fc367197e395de P a4532051e6c94424963f3862691fb2a4: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 23:10:59.000824 13253 tablet_bootstrap.cc:492] T c2646e3b08404b6180fc367197e395de P 5285a4ac2d32487890c7b633e69088fe: No bootstrap required, opened a new log
I20260620 23:10:59.001008 13253 ts_tablet_manager.cc:1397] T c2646e3b08404b6180fc367197e395de P 5285a4ac2d32487890c7b633e69088fe: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.010s
I20260620 23:10:59.001770 13253 raft_consensus.cc:348] T c2646e3b08404b6180fc367197e395de P 5285a4ac2d32487890c7b633e69088fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } }
I20260620 23:10:59.002027 13253 raft_consensus.cc:374] T c2646e3b08404b6180fc367197e395de P 5285a4ac2d32487890c7b633e69088fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:10:59.002147 13253 raft_consensus.cc:729] T c2646e3b08404b6180fc367197e395de P 5285a4ac2d32487890c7b633e69088fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5285a4ac2d32487890c7b633e69088fe, State: Initialized, Role: FOLLOWER
I20260620 23:10:59.002440 13253 consensus_queue.cc:260] T c2646e3b08404b6180fc367197e395de P 5285a4ac2d32487890c7b633e69088fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } }
I20260620 23:10:59.003106 13253 ts_tablet_manager.cc:1428] T c2646e3b08404b6180fc367197e395de P 5285a4ac2d32487890c7b633e69088fe: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 23:10:59.002946 13257 tablet_replica.cc:439] T c2646e3b08404b6180fc367197e395de P 5285a4ac2d32487890c7b633e69088fe: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } } }
I20260620 23:10:59.003336 13257 tablet_replica.cc:442] T c2646e3b08404b6180fc367197e395de P 5285a4ac2d32487890c7b633e69088fe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:10:59.011822 13259 raft_consensus.cc:482] T c2646e3b08404b6180fc367197e395de P a4532051e6c94424963f3862691fb2a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:10:59.012029 13259 raft_consensus.cc:504] T c2646e3b08404b6180fc367197e395de P a4532051e6c94424963f3862691fb2a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } }
I20260620 23:10:59.012681 13259 leader_election.cc:288] T c2646e3b08404b6180fc367197e395de P a4532051e6c94424963f3862691fb2a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 528ab34d28ba41f3b392a149fb596254 (127.10.191.194:45137), 5285a4ac2d32487890c7b633e69088fe (127.10.191.195:42307)
I20260620 23:10:59.013209 13043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2646e3b08404b6180fc367197e395de" candidate_uuid: "a4532051e6c94424963f3862691fb2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "528ab34d28ba41f3b392a149fb596254" is_pre_election: true
I20260620 23:10:59.013253 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2646e3b08404b6180fc367197e395de" candidate_uuid: "a4532051e6c94424963f3862691fb2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5285a4ac2d32487890c7b633e69088fe" is_pre_election: true
I20260620 23:10:59.013504 13043 raft_consensus.cc:2454] T c2646e3b08404b6180fc367197e395de P 528ab34d28ba41f3b392a149fb596254 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4532051e6c94424963f3862691fb2a4 in term 0.
I20260620 23:10:59.013535 13187 raft_consensus.cc:2454] T c2646e3b08404b6180fc367197e395de P 5285a4ac2d32487890c7b633e69088fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4532051e6c94424963f3862691fb2a4 in term 0.
I20260620 23:10:59.013911 12835 leader_election.cc:302] T c2646e3b08404b6180fc367197e395de P a4532051e6c94424963f3862691fb2a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 528ab34d28ba41f3b392a149fb596254, a4532051e6c94424963f3862691fb2a4; no voters:
I20260620 23:10:59.014206 13259 raft_consensus.cc:2781] T c2646e3b08404b6180fc367197e395de P a4532051e6c94424963f3862691fb2a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:10:59.014305 13259 raft_consensus.cc:482] T c2646e3b08404b6180fc367197e395de P a4532051e6c94424963f3862691fb2a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:10:59.014433 13259 raft_consensus.cc:3037] T c2646e3b08404b6180fc367197e395de P a4532051e6c94424963f3862691fb2a4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:10:59.016613 13259 raft_consensus.cc:504] T c2646e3b08404b6180fc367197e395de P a4532051e6c94424963f3862691fb2a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } }
I20260620 23:10:59.017246 13259 leader_election.cc:288] T c2646e3b08404b6180fc367197e395de P a4532051e6c94424963f3862691fb2a4 [CANDIDATE]: Term 1 election: Requested vote from peers 528ab34d28ba41f3b392a149fb596254 (127.10.191.194:45137), 5285a4ac2d32487890c7b633e69088fe (127.10.191.195:42307)
I20260620 23:10:59.017692 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2646e3b08404b6180fc367197e395de" candidate_uuid: "a4532051e6c94424963f3862691fb2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5285a4ac2d32487890c7b633e69088fe"
I20260620 23:10:59.017702 13043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2646e3b08404b6180fc367197e395de" candidate_uuid: "a4532051e6c94424963f3862691fb2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "528ab34d28ba41f3b392a149fb596254"
I20260620 23:10:59.017912 13187 raft_consensus.cc:3037] T c2646e3b08404b6180fc367197e395de P 5285a4ac2d32487890c7b633e69088fe [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:10:59.017912 13043 raft_consensus.cc:3037] T c2646e3b08404b6180fc367197e395de P 528ab34d28ba41f3b392a149fb596254 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:10:59.020875 13187 raft_consensus.cc:2454] T c2646e3b08404b6180fc367197e395de P 5285a4ac2d32487890c7b633e69088fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4532051e6c94424963f3862691fb2a4 in term 1.
I20260620 23:10:59.020875 13043 raft_consensus.cc:2454] T c2646e3b08404b6180fc367197e395de P 528ab34d28ba41f3b392a149fb596254 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4532051e6c94424963f3862691fb2a4 in term 1.
I20260620 23:10:59.021281 12835 leader_election.cc:302] T c2646e3b08404b6180fc367197e395de P a4532051e6c94424963f3862691fb2a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 528ab34d28ba41f3b392a149fb596254, a4532051e6c94424963f3862691fb2a4; no voters:
I20260620 23:10:59.021538 13259 raft_consensus.cc:2781] T c2646e3b08404b6180fc367197e395de P a4532051e6c94424963f3862691fb2a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:10:59.021721 13259 raft_consensus.cc:686] T c2646e3b08404b6180fc367197e395de P a4532051e6c94424963f3862691fb2a4 [term 1 LEADER]: Becoming Leader. State: Replica: a4532051e6c94424963f3862691fb2a4, State: Running, Role: LEADER
I20260620 23:10:59.021966 13259 consensus_queue.cc:237] T c2646e3b08404b6180fc367197e395de P a4532051e6c94424963f3862691fb2a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } }
I20260620 23:10:59.023706 13262 tablet_replica.cc:439] T c2646e3b08404b6180fc367197e395de P a4532051e6c94424963f3862691fb2a4: TxnStatusTablet state changed. Reason: New leader a4532051e6c94424963f3862691fb2a4. Latest consensus state: current_term: 1 leader_uuid: "a4532051e6c94424963f3862691fb2a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } } }
I20260620 23:10:59.024026 13262 tablet_replica.cc:442] T c2646e3b08404b6180fc367197e395de P a4532051e6c94424963f3862691fb2a4: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:10:59.025557 12746 catalog_manager.cc:5495] T c2646e3b08404b6180fc367197e395de P a4532051e6c94424963f3862691fb2a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4532051e6c94424963f3862691fb2a4 (127.10.191.193). New cstate: current_term: 1 leader_uuid: "a4532051e6c94424963f3862691fb2a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } health_report { overall_health: UNKNOWN } } }
I20260620 23:10:59.031384 13274 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 23:10:59.031683 13274 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 23:10:59.046476 13248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:59.040751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45898 (local address 127.10.191.193:40525)
0620 23:10:59.040934 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:59.040938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:59.040954 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:10:59.041046 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:59.041049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:59.041117 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:59.041242 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:10:59.041659 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:59.042391 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:59.044286 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:59.044724 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:59.044826 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:59.045421 (+ 595us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:10:59.045915 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:10:59.046031 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:59.046180 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 23:10:59.046248 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 23:10:59.046622 13269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:59.040463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:40525 (local address 127.0.0.1:45898)
0620 23:10:59.040710 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:59.040730 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:59.040868 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:59.041280 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:59.041285 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:59.041298 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:10:59.041491 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:59.041502 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:59.042526 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:59.042531 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:59.044119 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:59.044135 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:59.044889 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:59.044896 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:59.045127 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:59.045198 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:10:59.046035 (+ 837us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:10:59.046063 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 23:10:59.046157 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 23:10:59.046312 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 23:10:59.055186 13187 raft_consensus.cc:1261] T c2646e3b08404b6180fc367197e395de P 5285a4ac2d32487890c7b633e69088fe [term 1 FOLLOWER]: Refusing update from remote peer a4532051e6c94424963f3862691fb2a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:10:59.055271 13043 raft_consensus.cc:1261] T c2646e3b08404b6180fc367197e395de P 528ab34d28ba41f3b392a149fb596254 [term 1 FOLLOWER]: Refusing update from remote peer a4532051e6c94424963f3862691fb2a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:10:59.056177 13262 consensus_queue.cc:1035] T c2646e3b08404b6180fc367197e395de P a4532051e6c94424963f3862691fb2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:10:59.056612 13259 consensus_queue.cc:1035] T c2646e3b08404b6180fc367197e395de P a4532051e6c94424963f3862691fb2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:10:59.064071 13257 tablet_replica.cc:439] T c2646e3b08404b6180fc367197e395de P 5285a4ac2d32487890c7b633e69088fe: TxnStatusTablet state changed. Reason: New leader a4532051e6c94424963f3862691fb2a4. Latest consensus state: current_term: 1 leader_uuid: "a4532051e6c94424963f3862691fb2a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } } }
I20260620 23:10:59.064383 13257 tablet_replica.cc:442] T c2646e3b08404b6180fc367197e395de P 5285a4ac2d32487890c7b633e69088fe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:10:59.079689 13258 tablet_replica.cc:439] T c2646e3b08404b6180fc367197e395de P 528ab34d28ba41f3b392a149fb596254: TxnStatusTablet state changed. Reason: New leader a4532051e6c94424963f3862691fb2a4. Latest consensus state: current_term: 1 leader_uuid: "a4532051e6c94424963f3862691fb2a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } } }
I20260620 23:10:59.080008 13258 tablet_replica.cc:442] T c2646e3b08404b6180fc367197e395de P 528ab34d28ba41f3b392a149fb596254: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:10:59.081832 13257 tablet_replica.cc:439] T c2646e3b08404b6180fc367197e395de P 5285a4ac2d32487890c7b633e69088fe: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a4532051e6c94424963f3862691fb2a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } } }
I20260620 23:10:59.082072 13257 tablet_replica.cc:442] T c2646e3b08404b6180fc367197e395de P 5285a4ac2d32487890c7b633e69088fe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:10:59.084617 13259 tablet_replica.cc:439] T c2646e3b08404b6180fc367197e395de P a4532051e6c94424963f3862691fb2a4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a4532051e6c94424963f3862691fb2a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } } }
I20260620 23:10:59.084928 13259 tablet_replica.cc:442] T c2646e3b08404b6180fc367197e395de P a4532051e6c94424963f3862691fb2a4: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:10:59.087358 13259 tablet_replica.cc:439] T c2646e3b08404b6180fc367197e395de P a4532051e6c94424963f3862691fb2a4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a4532051e6c94424963f3862691fb2a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } } }
I20260620 23:10:59.087658 13259 tablet_replica.cc:442] T c2646e3b08404b6180fc367197e395de P a4532051e6c94424963f3862691fb2a4: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:10:59.091002 13258 tablet_replica.cc:439] T c2646e3b08404b6180fc367197e395de P 528ab34d28ba41f3b392a149fb596254: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a4532051e6c94424963f3862691fb2a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } } }
I20260620 23:10:59.091252 13258 tablet_replica.cc:442] T c2646e3b08404b6180fc367197e395de P 528ab34d28ba41f3b392a149fb596254: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:10:59.101603 13248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:59.096091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45912 (local address 127.10.191.193:40525)
0620 23:10:59.096303 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:59.096307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:59.096326 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:10:59.096373 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:59.096376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:59.096434 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:59.096505 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:10:59.096952 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:59.097725 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:59.099712 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:59.100138 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:59.100275 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:59.100736 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:10:59.100860 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:10:59.100932 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:59.101124 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 23:10:59.101192 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 23:10:59.121516 13248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:59.114174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:59869 (local address 127.10.191.193:40525)
0620 23:10:59.114328 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:59.114337 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:59.114994 (+ 657us) server_negotiation.cc:407] Connection header received
0620 23:10:59.115190 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:59.115196 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:59.115271 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:59.115372 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:10:59.116006 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:59.117124 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:59.119263 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:59.119980 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:59.120114 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:59.120723 (+ 609us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:10:59.120876 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:10:59.120967 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:59.121128 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 23:10:59.121212 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 23:10:59.121687 13288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:59.114083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:40525 (local address 127.10.191.193:59869)
0620 23:10:59.114887 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:59.114908 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:59.115055 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:59.115407 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:59.115413 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:59.115427 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:10:59.115845 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:59.115857 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:59.117287 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:59.117292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:59.119117 (+ 1825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:59.119131 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:59.120176 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:59.120183 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:59.120362 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:59.120493 (+ 131us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:10:59.120988 (+ 495us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:10:59.121002 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 23:10:59.121107 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 23:10:59.121261 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"mutex_wait_us":1,"thread_start_us":442,"threads_started":1}
I20260620 23:10:59.132171 13043 raft_consensus.cc:1261] T 956bab8764154b55abc7a626260480d4 P 528ab34d28ba41f3b392a149fb596254 [term 1 FOLLOWER]: Refusing update from remote peer a4532051e6c94424963f3862691fb2a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:10:59.132270 13187 raft_consensus.cc:1261] T 956bab8764154b55abc7a626260480d4 P 5285a4ac2d32487890c7b633e69088fe [term 1 FOLLOWER]: Refusing update from remote peer a4532051e6c94424963f3862691fb2a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:10:59.132992 13262 consensus_queue.cc:1035] T 956bab8764154b55abc7a626260480d4 P a4532051e6c94424963f3862691fb2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:10:59.133458 13262 consensus_queue.cc:1035] T 956bab8764154b55abc7a626260480d4 P a4532051e6c94424963f3862691fb2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:10:59.144627 13277 mvcc.cc:205] Tried to move back new op lower bound from 7299059954199142400 to 7299059953113223168. Current Snapshot: MvccSnapshot[applied={T|T < 7299059954199142400}]
I20260620 23:10:59.151329 13279 mvcc.cc:205] Tried to move back new op lower bound from 7299059954199142400 to 7299059953113223168. Current Snapshot: MvccSnapshot[applied={T|T < 7299059954199142400}]
I20260620 23:10:59.162204 13278 mvcc.cc:205] Tried to move back new op lower bound from 7299059954199142400 to 7299059953113223168. Current Snapshot: MvccSnapshot[applied={T|T < 7299059954199142400}]
I20260620 23:10:59.238408 12747 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:10:59.238677 12747 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:10:59.242329 12747 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 69ad19c2711d4c08b9b99bd0fe396704: Sending DeleteTablet for 3 replicas of tablet 956bab8764154b55abc7a626260480d4
I20260620 23:10:59.243610 13162 tablet_service.cc:1514] Processing DeleteTablet for tablet 956bab8764154b55abc7a626260480d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:10:59 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:43302
I20260620 23:10:59.243700 12883 tablet_service.cc:1514] Processing DeleteTablet for tablet 956bab8764154b55abc7a626260480d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:10:59 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:45896
I20260620 23:10:59.244472 13295 tablet_replica.cc:330] T 956bab8764154b55abc7a626260480d4 P 5285a4ac2d32487890c7b633e69088fe: stopping tablet replica
I20260620 23:10:59.244755 13296 tablet_replica.cc:330] T 956bab8764154b55abc7a626260480d4 P a4532051e6c94424963f3862691fb2a4: stopping tablet replica
I20260620 23:10:59.244915 13295 raft_consensus.cc:2229] T 956bab8764154b55abc7a626260480d4 P 5285a4ac2d32487890c7b633e69088fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:10:59.244982 13023 tablet_service.cc:1514] Processing DeleteTablet for tablet 956bab8764154b55abc7a626260480d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:10:59 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:51198
I20260620 23:10:59.245283 13295 raft_consensus.cc:2258] T 956bab8764154b55abc7a626260480d4 P 5285a4ac2d32487890c7b633e69088fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:10:59.245316 13296 raft_consensus.cc:2229] T 956bab8764154b55abc7a626260480d4 P a4532051e6c94424963f3862691fb2a4 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:10:59.246052 13296 raft_consensus.cc:2258] T 956bab8764154b55abc7a626260480d4 P a4532051e6c94424963f3862691fb2a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:10:59.247609 13295 ts_tablet_manager.cc:1905] T 956bab8764154b55abc7a626260480d4 P 5285a4ac2d32487890c7b633e69088fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:10:59.247888 13296 ts_tablet_manager.cc:1905] T 956bab8764154b55abc7a626260480d4 P a4532051e6c94424963f3862691fb2a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:10:59.250451 12746 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58166:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:10:59.251142 12746 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:10:59.254703 13295 ts_tablet_manager.cc:1918] T 956bab8764154b55abc7a626260480d4 P 5285a4ac2d32487890c7b633e69088fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:10:59.254858 13295 log.cc:1196] T 956bab8764154b55abc7a626260480d4 P 5285a4ac2d32487890c7b633e69088fe: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-2/wal/wals/956bab8764154b55abc7a626260480d4
I20260620 23:10:59.255865 13295 ts_tablet_manager.cc:1939] T 956bab8764154b55abc7a626260480d4 P 5285a4ac2d32487890c7b633e69088fe: Deleting consensus metadata
I20260620 23:10:59.256161 13296 ts_tablet_manager.cc:1918] T 956bab8764154b55abc7a626260480d4 P a4532051e6c94424963f3862691fb2a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:10:59.256403 13296 log.cc:1196] T 956bab8764154b55abc7a626260480d4 P a4532051e6c94424963f3862691fb2a4: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-0/wal/wals/956bab8764154b55abc7a626260480d4
I20260620 23:10:59.256965 13296 ts_tablet_manager.cc:1939] T 956bab8764154b55abc7a626260480d4 P a4532051e6c94424963f3862691fb2a4: Deleting consensus metadata
I20260620 23:10:59.257767 12733 catalog_manager.cc:4841] TS 5285a4ac2d32487890c7b633e69088fe (127.10.191.195:42307): tablet 956bab8764154b55abc7a626260480d4 (table test-table [id=3496e3835b484dd1ae2391912395dc1e]) successfully deleted
I20260620 23:10:59.258908 13162 tablet_service.cc:1467] Processing CreateTablet for tablet 08fa0c9cfb134c6eb5be3f8e05b1dbc0 (DEFAULT_TABLE table=test-table [id=96ef60d428664871a487b70144c14357]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:10:59.259127 13022 tablet_service.cc:1467] Processing CreateTablet for tablet 08fa0c9cfb134c6eb5be3f8e05b1dbc0 (DEFAULT_TABLE table=test-table [id=96ef60d428664871a487b70144c14357]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:10:59.259323 13162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08fa0c9cfb134c6eb5be3f8e05b1dbc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:59.259567 13022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08fa0c9cfb134c6eb5be3f8e05b1dbc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:59.263504 12732 catalog_manager.cc:4841] TS a4532051e6c94424963f3862691fb2a4 (127.10.191.193:40525): tablet 956bab8764154b55abc7a626260480d4 (table test-table [id=3496e3835b484dd1ae2391912395dc1e]) successfully deleted
I20260620 23:10:59.264045 12883 tablet_service.cc:1467] Processing CreateTablet for tablet 08fa0c9cfb134c6eb5be3f8e05b1dbc0 (DEFAULT_TABLE table=test-table [id=96ef60d428664871a487b70144c14357]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:10:59.265372 13253 tablet_bootstrap.cc:492] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 5285a4ac2d32487890c7b633e69088fe: Bootstrap starting.
I20260620 23:10:59.265677 13297 tablet_replica.cc:330] T 956bab8764154b55abc7a626260480d4 P 528ab34d28ba41f3b392a149fb596254: stopping tablet replica
I20260620 23:10:59.266278 13297 raft_consensus.cc:2229] T 956bab8764154b55abc7a626260480d4 P 528ab34d28ba41f3b392a149fb596254 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:10:59.266682 13297 raft_consensus.cc:2258] T 956bab8764154b55abc7a626260480d4 P 528ab34d28ba41f3b392a149fb596254 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:10:59.267050 13253 tablet_bootstrap.cc:654] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 5285a4ac2d32487890c7b633e69088fe: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:59.267838 13251 tablet_bootstrap.cc:492] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 528ab34d28ba41f3b392a149fb596254: Bootstrap starting.
I20260620 23:10:59.268666 13297 ts_tablet_manager.cc:1905] T 956bab8764154b55abc7a626260480d4 P 528ab34d28ba41f3b392a149fb596254: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:10:59.269448 13251 tablet_bootstrap.cc:654] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 528ab34d28ba41f3b392a149fb596254: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:59.271157 12883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08fa0c9cfb134c6eb5be3f8e05b1dbc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:10:59.278151 13297 ts_tablet_manager.cc:1918] T 956bab8764154b55abc7a626260480d4 P 528ab34d28ba41f3b392a149fb596254: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:10:59.278379 13297 log.cc:1196] T 956bab8764154b55abc7a626260480d4 P 528ab34d28ba41f3b392a149fb596254: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-1/wal/wals/956bab8764154b55abc7a626260480d4
I20260620 23:10:59.279007 13297 ts_tablet_manager.cc:1939] T 956bab8764154b55abc7a626260480d4 P 528ab34d28ba41f3b392a149fb596254: Deleting consensus metadata
I20260620 23:10:59.281579 13253 tablet_bootstrap.cc:492] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 5285a4ac2d32487890c7b633e69088fe: No bootstrap required, opened a new log
I20260620 23:10:59.281662 12732 catalog_manager.cc:4841] TS 528ab34d28ba41f3b392a149fb596254 (127.10.191.194:45137): tablet 956bab8764154b55abc7a626260480d4 (table test-table [id=3496e3835b484dd1ae2391912395dc1e]) successfully deleted
I20260620 23:10:59.281736 13251 tablet_bootstrap.cc:492] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 528ab34d28ba41f3b392a149fb596254: No bootstrap required, opened a new log
I20260620 23:10:59.281761 13253 ts_tablet_manager.cc:1397] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 5285a4ac2d32487890c7b633e69088fe: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.011s
I20260620 23:10:59.281905 13251 ts_tablet_manager.cc:1397] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 528ab34d28ba41f3b392a149fb596254: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.009s
I20260620 23:10:59.282523 13253 raft_consensus.cc:348] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 5285a4ac2d32487890c7b633e69088fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } }
I20260620 23:10:59.282807 13253 raft_consensus.cc:374] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 5285a4ac2d32487890c7b633e69088fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:10:59.282708 13251 raft_consensus.cc:348] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 528ab34d28ba41f3b392a149fb596254 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } }
I20260620 23:10:59.282902 13253 raft_consensus.cc:729] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 5285a4ac2d32487890c7b633e69088fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5285a4ac2d32487890c7b633e69088fe, State: Initialized, Role: FOLLOWER
I20260620 23:10:59.282996 13251 raft_consensus.cc:374] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 528ab34d28ba41f3b392a149fb596254 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:10:59.283079 13251 raft_consensus.cc:729] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 528ab34d28ba41f3b392a149fb596254 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 528ab34d28ba41f3b392a149fb596254, State: Initialized, Role: FOLLOWER
I20260620 23:10:59.283128 13253 consensus_queue.cc:260] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 5285a4ac2d32487890c7b633e69088fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } }
I20260620 23:10:59.283337 13251 consensus_queue.cc:260] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 528ab34d28ba41f3b392a149fb596254 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } }
I20260620 23:10:59.283689 13253 ts_tablet_manager.cc:1428] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 5285a4ac2d32487890c7b633e69088fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 23:10:59.285099 13251 ts_tablet_manager.cc:1428] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 528ab34d28ba41f3b392a149fb596254: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:10:59.286072 13252 tablet_bootstrap.cc:492] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P a4532051e6c94424963f3862691fb2a4: Bootstrap starting.
I20260620 23:10:59.287957 13252 tablet_bootstrap.cc:654] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P a4532051e6c94424963f3862691fb2a4: Neither blocks nor log segments found. Creating new log.
I20260620 23:10:59.290264 13252 tablet_bootstrap.cc:492] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P a4532051e6c94424963f3862691fb2a4: No bootstrap required, opened a new log
I20260620 23:10:59.290416 13252 ts_tablet_manager.cc:1397] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P a4532051e6c94424963f3862691fb2a4: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260620 23:10:59.291185 13252 raft_consensus.cc:348] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P a4532051e6c94424963f3862691fb2a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } }
I20260620 23:10:59.291404 13252 raft_consensus.cc:374] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P a4532051e6c94424963f3862691fb2a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:10:59.291484 13252 raft_consensus.cc:729] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P a4532051e6c94424963f3862691fb2a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4532051e6c94424963f3862691fb2a4, State: Initialized, Role: FOLLOWER
I20260620 23:10:59.291785 13252 consensus_queue.cc:260] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P a4532051e6c94424963f3862691fb2a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } }
I20260620 23:10:59.292357 13252 ts_tablet_manager.cc:1428] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P a4532051e6c94424963f3862691fb2a4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:10:59.508150 13258 raft_consensus.cc:482] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 528ab34d28ba41f3b392a149fb596254 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:10:59.508401 13258 raft_consensus.cc:504] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 528ab34d28ba41f3b392a149fb596254 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } }
I20260620 23:10:59.510291 13258 leader_election.cc:288] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 528ab34d28ba41f3b392a149fb596254 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4532051e6c94424963f3862691fb2a4 (127.10.191.193:40525), 5285a4ac2d32487890c7b633e69088fe (127.10.191.195:42307)
I20260620 23:10:59.530593 13248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:59.510901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:40487 (local address 127.10.191.193:40525)
0620 23:10:59.511088 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:59.511098 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:59.523275 (+ 12177us) server_negotiation.cc:407] Connection header received
0620 23:10:59.523609 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:59.523619 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:59.523737 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:59.523836 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:10:59.524643 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:59.525798 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:59.528170 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:59.529535 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:59.529720 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:59.529796 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:59.530158 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 23:10:59.530303 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 23:10:59.530764 13301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:59.510546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:40525 (local address 127.10.191.194:40487)
0620 23:10:59.523178 (+ 12632us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:59.523195 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:59.523404 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:59.524109 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:59.524117 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:59.524152 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:10:59.524444 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:59.524464 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:59.525937 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:59.525944 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:59.527973 (+ 2029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:59.527989 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:59.529714 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:59.529721 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:59.529957 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:59.529998 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 23:10:59.530103 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 23:10:59.530240 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12498,"mutex_wait_us":453,"thread_start_us":11534,"threads_started":1}
I20260620 23:10:59.531468 13302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:59.510748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:53535 (local address 127.10.191.195:42307)
0620 23:10:59.524522 (+ 13774us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:59.524528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:59.524553 (+ 25us) server_negotiation.cc:407] Connection header received
0620 23:10:59.524621 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:59.524625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:59.524700 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:59.524810 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:10:59.525481 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:59.526574 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:59.528906 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:59.530004 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:59.530166 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:59.530243 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:59.530942 (+ 699us) server_negotiation.cc:299] Negotiation successful
0620 23:10:59.531108 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13646,"thread_start_us":13538,"threads_started":1}
I20260620 23:10:59.531783 12904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08fa0c9cfb134c6eb5be3f8e05b1dbc0" candidate_uuid: "528ab34d28ba41f3b392a149fb596254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4532051e6c94424963f3862691fb2a4" is_pre_election: true
I20260620 23:10:59.532224 13303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:59.510546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.195:42307 (local address 127.10.191.194:53535)
0620 23:10:59.522802 (+ 12256us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:59.522831 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:59.523032 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:59.524851 (+ 1819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:59.524856 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:59.524883 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:10:59.525307 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:59.525320 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:59.526725 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:59.526733 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:59.528745 (+ 2012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:59.528756 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:59.530429 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:59.530436 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:59.530648 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:59.530703 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 23:10:59.531660 (+ 957us) client_negotiation.cc:240] Negotiation successful
0620 23:10:59.531815 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12102,"mutex_wait_us":55,"thread_start_us":12572,"threads_started":1}
I20260620 23:10:59.532297 12904 raft_consensus.cc:2454] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P a4532051e6c94424963f3862691fb2a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 528ab34d28ba41f3b392a149fb596254 in term 0.
I20260620 23:10:59.532964 12975 leader_election.cc:302] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 528ab34d28ba41f3b392a149fb596254 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 528ab34d28ba41f3b392a149fb596254, a4532051e6c94424963f3862691fb2a4; no voters:
I20260620 23:10:59.533179 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08fa0c9cfb134c6eb5be3f8e05b1dbc0" candidate_uuid: "528ab34d28ba41f3b392a149fb596254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5285a4ac2d32487890c7b633e69088fe" is_pre_election: true
I20260620 23:10:59.533411 13258 raft_consensus.cc:2781] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 528ab34d28ba41f3b392a149fb596254 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:10:59.533424 13187 raft_consensus.cc:2454] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 5285a4ac2d32487890c7b633e69088fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 528ab34d28ba41f3b392a149fb596254 in term 0.
I20260620 23:10:59.533527 13258 raft_consensus.cc:482] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 528ab34d28ba41f3b392a149fb596254 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:10:59.533613 13258 raft_consensus.cc:3037] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 528ab34d28ba41f3b392a149fb596254 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:10:59.536128 13258 raft_consensus.cc:504] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 528ab34d28ba41f3b392a149fb596254 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } }
I20260620 23:10:59.536726 13258 leader_election.cc:288] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 528ab34d28ba41f3b392a149fb596254 [CANDIDATE]: Term 1 election: Requested vote from peers a4532051e6c94424963f3862691fb2a4 (127.10.191.193:40525), 5285a4ac2d32487890c7b633e69088fe (127.10.191.195:42307)
I20260620 23:10:59.537127 12904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08fa0c9cfb134c6eb5be3f8e05b1dbc0" candidate_uuid: "528ab34d28ba41f3b392a149fb596254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4532051e6c94424963f3862691fb2a4"
I20260620 23:10:59.537222 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08fa0c9cfb134c6eb5be3f8e05b1dbc0" candidate_uuid: "528ab34d28ba41f3b392a149fb596254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5285a4ac2d32487890c7b633e69088fe"
I20260620 23:10:59.537400 12904 raft_consensus.cc:3037] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P a4532051e6c94424963f3862691fb2a4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:10:59.537431 13187 raft_consensus.cc:3037] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 5285a4ac2d32487890c7b633e69088fe [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:10:59.539958 12904 raft_consensus.cc:2454] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P a4532051e6c94424963f3862691fb2a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 528ab34d28ba41f3b392a149fb596254 in term 1.
I20260620 23:10:59.540352 12975 leader_election.cc:302] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 528ab34d28ba41f3b392a149fb596254 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 528ab34d28ba41f3b392a149fb596254, a4532051e6c94424963f3862691fb2a4; no voters:
I20260620 23:10:59.540489 13187 raft_consensus.cc:2454] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 5285a4ac2d32487890c7b633e69088fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 528ab34d28ba41f3b392a149fb596254 in term 1.
I20260620 23:10:59.540680 13258 raft_consensus.cc:2781] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 528ab34d28ba41f3b392a149fb596254 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:10:59.541254 13258 raft_consensus.cc:686] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 528ab34d28ba41f3b392a149fb596254 [term 1 LEADER]: Becoming Leader. State: Replica: 528ab34d28ba41f3b392a149fb596254, State: Running, Role: LEADER
I20260620 23:10:59.541917 13258 consensus_queue.cc:237] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 528ab34d28ba41f3b392a149fb596254 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } }
I20260620 23:10:59.545487 12747 catalog_manager.cc:5495] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 528ab34d28ba41f3b392a149fb596254 reported cstate change: term changed from 0 to 1, leader changed from <none> to 528ab34d28ba41f3b392a149fb596254 (127.10.191.194). New cstate: current_term: 1 leader_uuid: "528ab34d28ba41f3b392a149fb596254" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } health_report { overall_health: UNKNOWN } } }
I20260620 23:10:59.599830 13306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:59.593285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51202 (local address 127.10.191.194:45137)
0620 23:10:59.593878 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:59.593882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:59.593901 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:10:59.593975 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:59.593978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:59.594055 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:59.594190 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:10:59.594592 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:59.595451 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:59.597393 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:59.597775 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:59.597878 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:59.598390 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:10:59.599119 (+ 729us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:10:59.599269 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:59.599386 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 23:10:59.599465 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":319,"threads_started":1}
I20260620 23:10:59.643986 13248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:59.624663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:48355 (local address 127.10.191.193:40525)
0620 23:10:59.624859 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:59.624866 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:59.637391 (+ 12525us) server_negotiation.cc:407] Connection header received
0620 23:10:59.637639 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:59.637644 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:59.637728 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:59.637818 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:10:59.638536 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:59.639728 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:59.641762 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:59.642445 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:59.642609 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:59.643174 (+ 565us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:10:59.643343 (+ 169us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:10:59.643440 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:59.643583 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 23:10:59.643672 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 23:10:59.644083 13309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:59.624681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:40525 (local address 127.10.191.194:48355)
0620 23:10:59.637246 (+ 12565us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:59.637264 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:59.637494 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:59.638008 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:59.638013 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:59.638026 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:10:59.638360 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:59.638371 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:59.639922 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:59.639927 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:59.641589 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:59.641604 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:59.642664 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:59.642683 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:59.642901 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:59.642977 (+ 76us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:10:59.643453 (+ 476us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:10:59.643465 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 23:10:59.643544 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 23:10:59.643694 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12388,"thread_start_us":12204,"threads_started":1}
I20260620 23:10:59.653712 12904 raft_consensus.cc:1261] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P a4532051e6c94424963f3862691fb2a4 [term 1 FOLLOWER]: Refusing update from remote peer 528ab34d28ba41f3b392a149fb596254: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:10:59.654523 13187 raft_consensus.cc:1261] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 5285a4ac2d32487890c7b633e69088fe [term 1 FOLLOWER]: Refusing update from remote peer 528ab34d28ba41f3b392a149fb596254: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:10:59.654570 13304 consensus_queue.cc:1035] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 528ab34d28ba41f3b392a149fb596254 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:10:59.655092 13258 consensus_queue.cc:1035] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 528ab34d28ba41f3b392a149fb596254 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:10:59.691416 13306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:59.685537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:54207 (local address 127.10.191.194:45137)
0620 23:10:59.685716 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 23:10:59.685722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:10:59.685927 (+ 205us) server_negotiation.cc:407] Connection header received
0620 23:10:59.686138 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:10:59.686143 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:10:59.686217 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:10:59.686304 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:10:59.686819 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:59.687996 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:59.689450 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:10:59.690081 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:10:59.690207 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:59.690656 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:10:59.690794 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:10:59.690915 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 23:10:59.691051 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 23:10:59.691137 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 23:10:59.691496 13314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:10:59.685262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.194:45137 (local address 127.10.191.193:54207)
0620 23:10:59.685841 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0620 23:10:59.685860 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:10:59.685993 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:10:59.686342 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:10:59.686348 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:10:59.686361 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:10:59.686646 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:59.686658 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:59.688128 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:59.688132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:59.689311 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:10:59.689323 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:10:59.690223 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:10:59.690229 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:10:59.690437 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:10:59.690488 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:10:59.690919 (+ 431us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:10:59.690929 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 23:10:59.691018 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 23:10:59.691174 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":461,"threads_started":1}
W20260620 23:10:59.692899 13279 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 23:11:01.291616 12747 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:11:01.291850 12747 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:11:01.295667 12747 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 69ad19c2711d4c08b9b99bd0fe396704: Sending DeleteTablet for 3 replicas of tablet 08fa0c9cfb134c6eb5be3f8e05b1dbc0
I20260620 23:11:01.296407 12883 tablet_service.cc:1514] Processing DeleteTablet for tablet 08fa0c9cfb134c6eb5be3f8e05b1dbc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:11:01 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:45896
I20260620 23:11:01.296490 13022 tablet_service.cc:1514] Processing DeleteTablet for tablet 08fa0c9cfb134c6eb5be3f8e05b1dbc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:11:01 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:51198
I20260620 23:11:01.296929 13162 tablet_service.cc:1514] Processing DeleteTablet for tablet 08fa0c9cfb134c6eb5be3f8e05b1dbc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:11:01 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:43302
I20260620 23:11:01.297078 13338 tablet_replica.cc:330] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P a4532051e6c94424963f3862691fb2a4: stopping tablet replica
I20260620 23:11:01.297276 13338 raft_consensus.cc:2229] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P a4532051e6c94424963f3862691fb2a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:11:01.297377 13339 tablet_replica.cc:330] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 528ab34d28ba41f3b392a149fb596254: stopping tablet replica
I20260620 23:11:01.297586 13338 raft_consensus.cc:2258] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P a4532051e6c94424963f3862691fb2a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:11:01.297609 13339 raft_consensus.cc:2229] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 528ab34d28ba41f3b392a149fb596254 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:11:01.298223 13339 raft_consensus.cc:2258] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 528ab34d28ba41f3b392a149fb596254 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:11:01.299031 13338 ts_tablet_manager.cc:1905] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P a4532051e6c94424963f3862691fb2a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:11:01.299118 13339 ts_tablet_manager.cc:1905] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 528ab34d28ba41f3b392a149fb596254: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:11:01.305761 13339 ts_tablet_manager.cc:1918] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 528ab34d28ba41f3b392a149fb596254: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 23:11:01.305860 13338 ts_tablet_manager.cc:1918] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P a4532051e6c94424963f3862691fb2a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 23:11:01.306057 13338 log.cc:1196] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P a4532051e6c94424963f3862691fb2a4: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-0/wal/wals/08fa0c9cfb134c6eb5be3f8e05b1dbc0
I20260620 23:11:01.306245 13340 tablet_replica.cc:330] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 5285a4ac2d32487890c7b633e69088fe: stopping tablet replica
I20260620 23:11:01.306478 13340 raft_consensus.cc:2229] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 5285a4ac2d32487890c7b633e69088fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:11:01.306532 13338 ts_tablet_manager.cc:1939] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P a4532051e6c94424963f3862691fb2a4: Deleting consensus metadata
I20260620 23:11:01.306738 13340 raft_consensus.cc:2258] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 5285a4ac2d32487890c7b633e69088fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:11:01.307562 13339 log.cc:1196] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 528ab34d28ba41f3b392a149fb596254: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-1/wal/wals/08fa0c9cfb134c6eb5be3f8e05b1dbc0
I20260620 23:11:01.307729 13340 ts_tablet_manager.cc:1905] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 5285a4ac2d32487890c7b633e69088fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:11:01.307777 12732 catalog_manager.cc:4841] TS a4532051e6c94424963f3862691fb2a4 (127.10.191.193:40525): tablet 08fa0c9cfb134c6eb5be3f8e05b1dbc0 (table test-table [id=96ef60d428664871a487b70144c14357]) successfully deleted
I20260620 23:11:01.308121 13339 ts_tablet_manager.cc:1939] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 528ab34d28ba41f3b392a149fb596254: Deleting consensus metadata
I20260620 23:11:01.309197 12732 catalog_manager.cc:4841] TS 528ab34d28ba41f3b392a149fb596254 (127.10.191.194:45137): tablet 08fa0c9cfb134c6eb5be3f8e05b1dbc0 (table test-table [id=96ef60d428664871a487b70144c14357]) successfully deleted
I20260620 23:11:01.312405 13340 ts_tablet_manager.cc:1918] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 5285a4ac2d32487890c7b633e69088fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 23:11:01.312546 13340 log.cc:1196] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 5285a4ac2d32487890c7b633e69088fe: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-2/wal/wals/08fa0c9cfb134c6eb5be3f8e05b1dbc0
I20260620 23:11:01.312971 13340 ts_tablet_manager.cc:1939] T 08fa0c9cfb134c6eb5be3f8e05b1dbc0 P 5285a4ac2d32487890c7b633e69088fe: Deleting consensus metadata
I20260620 23:11:01.313851 12733 catalog_manager.cc:4841] TS 5285a4ac2d32487890c7b633e69088fe (127.10.191.195:42307): tablet 08fa0c9cfb134c6eb5be3f8e05b1dbc0 (table test-table [id=96ef60d428664871a487b70144c14357]) successfully deleted
Jun 20 23:11:01 dist-test-slave-jkbq krb5kdc[12702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997061, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 23:11:01 dist-test-slave-jkbq krb5kdc[12702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997061, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:11:01.354451 13348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:01.334944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58192 (local address 127.10.191.254:36523)
0620 23:11:01.335468 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:01.335473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:01.335551 (+ 78us) server_negotiation.cc:407] Connection header received
0620 23:11:01.335720 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:01.335724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:01.335800 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:01.335901 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:01.337281 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:01.338090 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:01.339499 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:01.339964 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:01.340061 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:01.349313 (+ 9252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:01.349318 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:01.349321 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:01.349360 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:01.351861 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:01.352526 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:01.352532 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:01.352536 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:01.352663 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:01.353036 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:01.353041 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:01.353044 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:01.353323 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:01.353568 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:01.353867 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 23:11:01.354088 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":271,"threads_started":1}
I20260620 23:11:01.357942 12747 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:11:01.358546 12747 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:11:01.364777 13022 tablet_service.cc:1467] Processing CreateTablet for tablet 41d989b6b1de4388a803fe0351cbb8fd (DEFAULT_TABLE table=test-table [id=01bea322d2a24a1dbf955e7ce9b38a2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:11:01.364774 12883 tablet_service.cc:1467] Processing CreateTablet for tablet 41d989b6b1de4388a803fe0351cbb8fd (DEFAULT_TABLE table=test-table [id=01bea322d2a24a1dbf955e7ce9b38a2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:11:01.365028 13162 tablet_service.cc:1467] Processing CreateTablet for tablet 41d989b6b1de4388a803fe0351cbb8fd (DEFAULT_TABLE table=test-table [id=01bea322d2a24a1dbf955e7ce9b38a2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:11:01.365218 12883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41d989b6b1de4388a803fe0351cbb8fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:11:01.365231 13022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41d989b6b1de4388a803fe0351cbb8fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:11:01.365432 13162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41d989b6b1de4388a803fe0351cbb8fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:11:01.371348 13349 tablet_bootstrap.cc:492] T 41d989b6b1de4388a803fe0351cbb8fd P a4532051e6c94424963f3862691fb2a4: Bootstrap starting.
I20260620 23:11:01.372113 13350 tablet_bootstrap.cc:492] T 41d989b6b1de4388a803fe0351cbb8fd P 5285a4ac2d32487890c7b633e69088fe: Bootstrap starting.
I20260620 23:11:01.372884 13351 tablet_bootstrap.cc:492] T 41d989b6b1de4388a803fe0351cbb8fd P 528ab34d28ba41f3b392a149fb596254: Bootstrap starting.
I20260620 23:11:01.373145 13349 tablet_bootstrap.cc:654] T 41d989b6b1de4388a803fe0351cbb8fd P a4532051e6c94424963f3862691fb2a4: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:01.373934 13350 tablet_bootstrap.cc:654] T 41d989b6b1de4388a803fe0351cbb8fd P 5285a4ac2d32487890c7b633e69088fe: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:01.374531 13351 tablet_bootstrap.cc:654] T 41d989b6b1de4388a803fe0351cbb8fd P 528ab34d28ba41f3b392a149fb596254: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:01.375810 13349 tablet_bootstrap.cc:492] T 41d989b6b1de4388a803fe0351cbb8fd P a4532051e6c94424963f3862691fb2a4: No bootstrap required, opened a new log
I20260620 23:11:01.375976 13349 ts_tablet_manager.cc:1397] T 41d989b6b1de4388a803fe0351cbb8fd P a4532051e6c94424963f3862691fb2a4: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260620 23:11:01.376613 13350 tablet_bootstrap.cc:492] T 41d989b6b1de4388a803fe0351cbb8fd P 5285a4ac2d32487890c7b633e69088fe: No bootstrap required, opened a new log
I20260620 23:11:01.376705 13351 tablet_bootstrap.cc:492] T 41d989b6b1de4388a803fe0351cbb8fd P 528ab34d28ba41f3b392a149fb596254: No bootstrap required, opened a new log
I20260620 23:11:01.376765 13350 ts_tablet_manager.cc:1397] T 41d989b6b1de4388a803fe0351cbb8fd P 5285a4ac2d32487890c7b633e69088fe: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 23:11:01.376849 13351 ts_tablet_manager.cc:1397] T 41d989b6b1de4388a803fe0351cbb8fd P 528ab34d28ba41f3b392a149fb596254: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260620 23:11:01.376768 13349 raft_consensus.cc:348] T 41d989b6b1de4388a803fe0351cbb8fd P a4532051e6c94424963f3862691fb2a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } }
I20260620 23:11:01.377041 13349 raft_consensus.cc:374] T 41d989b6b1de4388a803fe0351cbb8fd P a4532051e6c94424963f3862691fb2a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:01.377142 13349 raft_consensus.cc:729] T 41d989b6b1de4388a803fe0351cbb8fd P a4532051e6c94424963f3862691fb2a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4532051e6c94424963f3862691fb2a4, State: Initialized, Role: FOLLOWER
I20260620 23:11:01.377367 13349 consensus_queue.cc:260] T 41d989b6b1de4388a803fe0351cbb8fd P a4532051e6c94424963f3862691fb2a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } }
I20260620 23:11:01.377574 13350 raft_consensus.cc:348] T 41d989b6b1de4388a803fe0351cbb8fd P 5285a4ac2d32487890c7b633e69088fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } }
I20260620 23:11:01.377619 13351 raft_consensus.cc:348] T 41d989b6b1de4388a803fe0351cbb8fd P 528ab34d28ba41f3b392a149fb596254 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } }
I20260620 23:11:01.377810 13350 raft_consensus.cc:374] T 41d989b6b1de4388a803fe0351cbb8fd P 5285a4ac2d32487890c7b633e69088fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:01.377852 13351 raft_consensus.cc:374] T 41d989b6b1de4388a803fe0351cbb8fd P 528ab34d28ba41f3b392a149fb596254 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:01.377905 13350 raft_consensus.cc:729] T 41d989b6b1de4388a803fe0351cbb8fd P 5285a4ac2d32487890c7b633e69088fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5285a4ac2d32487890c7b633e69088fe, State: Initialized, Role: FOLLOWER
I20260620 23:11:01.377902 13349 ts_tablet_manager.cc:1428] T 41d989b6b1de4388a803fe0351cbb8fd P a4532051e6c94424963f3862691fb2a4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 23:11:01.377945 13351 raft_consensus.cc:729] T 41d989b6b1de4388a803fe0351cbb8fd P 528ab34d28ba41f3b392a149fb596254 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 528ab34d28ba41f3b392a149fb596254, State: Initialized, Role: FOLLOWER
I20260620 23:11:01.378129 13350 consensus_queue.cc:260] T 41d989b6b1de4388a803fe0351cbb8fd P 5285a4ac2d32487890c7b633e69088fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } }
I20260620 23:11:01.378196 13351 consensus_queue.cc:260] T 41d989b6b1de4388a803fe0351cbb8fd P 528ab34d28ba41f3b392a149fb596254 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } }
I20260620 23:11:01.379271 13350 ts_tablet_manager.cc:1428] T 41d989b6b1de4388a803fe0351cbb8fd P 5285a4ac2d32487890c7b633e69088fe: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 23:11:01.379626 13351 ts_tablet_manager.cc:1428] T 41d989b6b1de4388a803fe0351cbb8fd P 528ab34d28ba41f3b392a149fb596254: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 23:11:01.442701 13329 raft_consensus.cc:482] T 41d989b6b1de4388a803fe0351cbb8fd P 528ab34d28ba41f3b392a149fb596254 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:11:01.442979 13329 raft_consensus.cc:504] T 41d989b6b1de4388a803fe0351cbb8fd P 528ab34d28ba41f3b392a149fb596254 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } }
I20260620 23:11:01.443605 13329 leader_election.cc:288] T 41d989b6b1de4388a803fe0351cbb8fd P 528ab34d28ba41f3b392a149fb596254 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4532051e6c94424963f3862691fb2a4 (127.10.191.193:40525), 5285a4ac2d32487890c7b633e69088fe (127.10.191.195:42307)
I20260620 23:11:01.444271 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41d989b6b1de4388a803fe0351cbb8fd" candidate_uuid: "528ab34d28ba41f3b392a149fb596254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5285a4ac2d32487890c7b633e69088fe" is_pre_election: true
I20260620 23:11:01.444257 12904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41d989b6b1de4388a803fe0351cbb8fd" candidate_uuid: "528ab34d28ba41f3b392a149fb596254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4532051e6c94424963f3862691fb2a4" is_pre_election: true
I20260620 23:11:01.444543 12904 raft_consensus.cc:2454] T 41d989b6b1de4388a803fe0351cbb8fd P a4532051e6c94424963f3862691fb2a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 528ab34d28ba41f3b392a149fb596254 in term 0.
I20260620 23:11:01.444548 13187 raft_consensus.cc:2454] T 41d989b6b1de4388a803fe0351cbb8fd P 5285a4ac2d32487890c7b633e69088fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 528ab34d28ba41f3b392a149fb596254 in term 0.
I20260620 23:11:01.445061 12976 leader_election.cc:302] T 41d989b6b1de4388a803fe0351cbb8fd P 528ab34d28ba41f3b392a149fb596254 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5285a4ac2d32487890c7b633e69088fe, 528ab34d28ba41f3b392a149fb596254; no voters:
I20260620 23:11:01.445356 13329 raft_consensus.cc:2781] T 41d989b6b1de4388a803fe0351cbb8fd P 528ab34d28ba41f3b392a149fb596254 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:11:01.445459 13329 raft_consensus.cc:482] T 41d989b6b1de4388a803fe0351cbb8fd P 528ab34d28ba41f3b392a149fb596254 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:11:01.445567 13329 raft_consensus.cc:3037] T 41d989b6b1de4388a803fe0351cbb8fd P 528ab34d28ba41f3b392a149fb596254 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:01.447568 13329 raft_consensus.cc:504] T 41d989b6b1de4388a803fe0351cbb8fd P 528ab34d28ba41f3b392a149fb596254 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } }
I20260620 23:11:01.448136 13329 leader_election.cc:288] T 41d989b6b1de4388a803fe0351cbb8fd P 528ab34d28ba41f3b392a149fb596254 [CANDIDATE]: Term 1 election: Requested vote from peers a4532051e6c94424963f3862691fb2a4 (127.10.191.193:40525), 5285a4ac2d32487890c7b633e69088fe (127.10.191.195:42307)
I20260620 23:11:01.448582 12904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41d989b6b1de4388a803fe0351cbb8fd" candidate_uuid: "528ab34d28ba41f3b392a149fb596254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4532051e6c94424963f3862691fb2a4"
I20260620 23:11:01.448668 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41d989b6b1de4388a803fe0351cbb8fd" candidate_uuid: "528ab34d28ba41f3b392a149fb596254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5285a4ac2d32487890c7b633e69088fe"
I20260620 23:11:01.448757 12904 raft_consensus.cc:3037] T 41d989b6b1de4388a803fe0351cbb8fd P a4532051e6c94424963f3862691fb2a4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:01.448897 13187 raft_consensus.cc:3037] T 41d989b6b1de4388a803fe0351cbb8fd P 5285a4ac2d32487890c7b633e69088fe [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:01.450719 12904 raft_consensus.cc:2454] T 41d989b6b1de4388a803fe0351cbb8fd P a4532051e6c94424963f3862691fb2a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 528ab34d28ba41f3b392a149fb596254 in term 1.
I20260620 23:11:01.450821 13187 raft_consensus.cc:2454] T 41d989b6b1de4388a803fe0351cbb8fd P 5285a4ac2d32487890c7b633e69088fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 528ab34d28ba41f3b392a149fb596254 in term 1.
I20260620 23:11:01.451135 12975 leader_election.cc:302] T 41d989b6b1de4388a803fe0351cbb8fd P 528ab34d28ba41f3b392a149fb596254 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 528ab34d28ba41f3b392a149fb596254, a4532051e6c94424963f3862691fb2a4; no voters:
I20260620 23:11:01.451444 13329 raft_consensus.cc:2781] T 41d989b6b1de4388a803fe0351cbb8fd P 528ab34d28ba41f3b392a149fb596254 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:11:01.451591 13329 raft_consensus.cc:686] T 41d989b6b1de4388a803fe0351cbb8fd P 528ab34d28ba41f3b392a149fb596254 [term 1 LEADER]: Becoming Leader. State: Replica: 528ab34d28ba41f3b392a149fb596254, State: Running, Role: LEADER
I20260620 23:11:01.451854 13329 consensus_queue.cc:237] T 41d989b6b1de4388a803fe0351cbb8fd P 528ab34d28ba41f3b392a149fb596254 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } }
I20260620 23:11:01.454416 12746 catalog_manager.cc:5495] T 41d989b6b1de4388a803fe0351cbb8fd P 528ab34d28ba41f3b392a149fb596254 reported cstate change: term changed from 0 to 1, leader changed from <none> to 528ab34d28ba41f3b392a149fb596254 (127.10.191.194). New cstate: current_term: 1 leader_uuid: "528ab34d28ba41f3b392a149fb596254" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } health_report { overall_health: UNKNOWN } } }
I20260620 23:11:01.480544 13358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:01.475707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51212 (local address 127.10.191.194:45137)
0620 23:11:01.476301 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:01.476305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:01.476324 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:11:01.476376 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:01.476379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:01.476435 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:01.476516 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:11:01.476806 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:01.477541 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:01.478840 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:01.479384 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:01.479481 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:01.479861 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:11:01.479968 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:11:01.480029 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:01.480219 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 23:11:01.480287 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":361,"threads_started":1}
I20260620 23:11:01.488776 12904 raft_consensus.cc:1261] T 41d989b6b1de4388a803fe0351cbb8fd P a4532051e6c94424963f3862691fb2a4 [term 1 FOLLOWER]: Refusing update from remote peer 528ab34d28ba41f3b392a149fb596254: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:11:01.488826 13187 raft_consensus.cc:1261] T 41d989b6b1de4388a803fe0351cbb8fd P 5285a4ac2d32487890c7b633e69088fe [term 1 FOLLOWER]: Refusing update from remote peer 528ab34d28ba41f3b392a149fb596254: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:11:01.489315 13258 consensus_queue.cc:1035] T 41d989b6b1de4388a803fe0351cbb8fd P 528ab34d28ba41f3b392a149fb596254 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:11:01.489563 13329 consensus_queue.cc:1035] T 41d989b6b1de4388a803fe0351cbb8fd P 528ab34d28ba41f3b392a149fb596254 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:11:01.531507 12745 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:11:01.531776 12745 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:11:01.535070 12745 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 69ad19c2711d4c08b9b99bd0fe396704: Sending DeleteTablet for 3 replicas of tablet 41d989b6b1de4388a803fe0351cbb8fd
I20260620 23:11:01.535853 13162 tablet_service.cc:1514] Processing DeleteTablet for tablet 41d989b6b1de4388a803fe0351cbb8fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:11:01 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:43302
I20260620 23:11:01.535853 12883 tablet_service.cc:1514] Processing DeleteTablet for tablet 41d989b6b1de4388a803fe0351cbb8fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:11:01 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:45896
I20260620 23:11:01.536125 13340 tablet_replica.cc:330] T 41d989b6b1de4388a803fe0351cbb8fd P 5285a4ac2d32487890c7b633e69088fe: stopping tablet replica
I20260620 23:11:01.536139 13022 tablet_service.cc:1514] Processing DeleteTablet for tablet 41d989b6b1de4388a803fe0351cbb8fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:11:01 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:51198
I20260620 23:11:01.536334 13340 raft_consensus.cc:2229] T 41d989b6b1de4388a803fe0351cbb8fd P 5285a4ac2d32487890c7b633e69088fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:11:01.536409 13338 tablet_replica.cc:330] T 41d989b6b1de4388a803fe0351cbb8fd P a4532051e6c94424963f3862691fb2a4: stopping tablet replica
I20260620 23:11:01.536394 13339 tablet_replica.cc:330] T 41d989b6b1de4388a803fe0351cbb8fd P 528ab34d28ba41f3b392a149fb596254: stopping tablet replica
I20260620 23:11:01.536530 13340 raft_consensus.cc:2258] T 41d989b6b1de4388a803fe0351cbb8fd P 5285a4ac2d32487890c7b633e69088fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:11:01.536593 13338 raft_consensus.cc:2229] T 41d989b6b1de4388a803fe0351cbb8fd P a4532051e6c94424963f3862691fb2a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:11:01.536602 13339 raft_consensus.cc:2229] T 41d989b6b1de4388a803fe0351cbb8fd P 528ab34d28ba41f3b392a149fb596254 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:11:01.536753 13338 raft_consensus.cc:2258] T 41d989b6b1de4388a803fe0351cbb8fd P a4532051e6c94424963f3862691fb2a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:11:01.536903 13339 raft_consensus.cc:2258] T 41d989b6b1de4388a803fe0351cbb8fd P 528ab34d28ba41f3b392a149fb596254 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:11:01.536855 12745 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:11:01.537544 12745 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:11:01.537595 13340 ts_tablet_manager.cc:1905] T 41d989b6b1de4388a803fe0351cbb8fd P 5285a4ac2d32487890c7b633e69088fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:11:01.537696 13338 ts_tablet_manager.cc:1905] T 41d989b6b1de4388a803fe0351cbb8fd P a4532051e6c94424963f3862691fb2a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:11:01.538115 13339 ts_tablet_manager.cc:1905] T 41d989b6b1de4388a803fe0351cbb8fd P 528ab34d28ba41f3b392a149fb596254: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:11:01.544184 13340 ts_tablet_manager.cc:1918] T 41d989b6b1de4388a803fe0351cbb8fd P 5285a4ac2d32487890c7b633e69088fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:11:01.544184 13338 ts_tablet_manager.cc:1918] T 41d989b6b1de4388a803fe0351cbb8fd P a4532051e6c94424963f3862691fb2a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:11:01.544348 13340 log.cc:1196] T 41d989b6b1de4388a803fe0351cbb8fd P 5285a4ac2d32487890c7b633e69088fe: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-2/wal/wals/41d989b6b1de4388a803fe0351cbb8fd
I20260620 23:11:01.544585 13339 ts_tablet_manager.cc:1918] T 41d989b6b1de4388a803fe0351cbb8fd P 528ab34d28ba41f3b392a149fb596254: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:11:01.544812 13022 tablet_service.cc:1467] Processing CreateTablet for tablet 72b6a646820a445c8eedcfcb62936d6a (DEFAULT_TABLE table=test-table [id=6274c724193b4c49a2dc9873b4c59e08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:11:01.545264 13022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72b6a646820a445c8eedcfcb62936d6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:11:01.545243 13162 tablet_service.cc:1467] Processing CreateTablet for tablet 72b6a646820a445c8eedcfcb62936d6a (DEFAULT_TABLE table=test-table [id=6274c724193b4c49a2dc9873b4c59e08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:11:01.545627 13162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72b6a646820a445c8eedcfcb62936d6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:11:01.551702 13350 tablet_bootstrap.cc:492] T 72b6a646820a445c8eedcfcb62936d6a P 5285a4ac2d32487890c7b633e69088fe: Bootstrap starting.
I20260620 23:11:01.551834 13351 tablet_bootstrap.cc:492] T 72b6a646820a445c8eedcfcb62936d6a P 528ab34d28ba41f3b392a149fb596254: Bootstrap starting.
I20260620 23:11:01.551256 12883 tablet_service.cc:1467] Processing CreateTablet for tablet 72b6a646820a445c8eedcfcb62936d6a (DEFAULT_TABLE table=test-table [id=6274c724193b4c49a2dc9873b4c59e08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:11:01.552265 12883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72b6a646820a445c8eedcfcb62936d6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:11:01.553050 13350 tablet_bootstrap.cc:654] T 72b6a646820a445c8eedcfcb62936d6a P 5285a4ac2d32487890c7b633e69088fe: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:01.553350 13351 tablet_bootstrap.cc:654] T 72b6a646820a445c8eedcfcb62936d6a P 528ab34d28ba41f3b392a149fb596254: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:01.551834 13339 log.cc:1196] T 41d989b6b1de4388a803fe0351cbb8fd P 528ab34d28ba41f3b392a149fb596254: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-1/wal/wals/41d989b6b1de4388a803fe0351cbb8fd
I20260620 23:11:01.555517 13339 ts_tablet_manager.cc:1939] T 41d989b6b1de4388a803fe0351cbb8fd P 528ab34d28ba41f3b392a149fb596254: Deleting consensus metadata
I20260620 23:11:01.556805 12732 catalog_manager.cc:4841] TS 528ab34d28ba41f3b392a149fb596254 (127.10.191.194:45137): tablet 41d989b6b1de4388a803fe0351cbb8fd (table test-table [id=01bea322d2a24a1dbf955e7ce9b38a2e]) successfully deleted
I20260620 23:11:01.558254 13340 ts_tablet_manager.cc:1939] T 41d989b6b1de4388a803fe0351cbb8fd P 5285a4ac2d32487890c7b633e69088fe: Deleting consensus metadata
I20260620 23:11:01.544349 13338 log.cc:1196] T 41d989b6b1de4388a803fe0351cbb8fd P a4532051e6c94424963f3862691fb2a4: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-0/wal/wals/41d989b6b1de4388a803fe0351cbb8fd
I20260620 23:11:01.559445 13338 ts_tablet_manager.cc:1939] T 41d989b6b1de4388a803fe0351cbb8fd P a4532051e6c94424963f3862691fb2a4: Deleting consensus metadata
I20260620 23:11:01.559517 12733 catalog_manager.cc:4841] TS 5285a4ac2d32487890c7b633e69088fe (127.10.191.195:42307): tablet 41d989b6b1de4388a803fe0351cbb8fd (table test-table [id=01bea322d2a24a1dbf955e7ce9b38a2e]) successfully deleted
I20260620 23:11:01.560750 12732 catalog_manager.cc:4841] TS a4532051e6c94424963f3862691fb2a4 (127.10.191.193:40525): tablet 41d989b6b1de4388a803fe0351cbb8fd (table test-table [id=01bea322d2a24a1dbf955e7ce9b38a2e]) successfully deleted
I20260620 23:11:01.561635 13349 tablet_bootstrap.cc:492] T 72b6a646820a445c8eedcfcb62936d6a P a4532051e6c94424963f3862691fb2a4: Bootstrap starting.
I20260620 23:11:01.563830 13349 tablet_bootstrap.cc:654] T 72b6a646820a445c8eedcfcb62936d6a P a4532051e6c94424963f3862691fb2a4: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:01.564957 13350 tablet_bootstrap.cc:492] T 72b6a646820a445c8eedcfcb62936d6a P 5285a4ac2d32487890c7b633e69088fe: No bootstrap required, opened a new log
I20260620 23:11:01.565173 13350 ts_tablet_manager.cc:1397] T 72b6a646820a445c8eedcfcb62936d6a P 5285a4ac2d32487890c7b633e69088fe: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.008s
I20260620 23:11:01.565953 13351 tablet_bootstrap.cc:492] T 72b6a646820a445c8eedcfcb62936d6a P 528ab34d28ba41f3b392a149fb596254: No bootstrap required, opened a new log
I20260620 23:11:01.565864 13350 raft_consensus.cc:348] T 72b6a646820a445c8eedcfcb62936d6a P 5285a4ac2d32487890c7b633e69088fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } }
I20260620 23:11:01.566094 13350 raft_consensus.cc:374] T 72b6a646820a445c8eedcfcb62936d6a P 5285a4ac2d32487890c7b633e69088fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:01.566140 13351 ts_tablet_manager.cc:1397] T 72b6a646820a445c8eedcfcb62936d6a P 528ab34d28ba41f3b392a149fb596254: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.013s
I20260620 23:11:01.566182 13350 raft_consensus.cc:729] T 72b6a646820a445c8eedcfcb62936d6a P 5285a4ac2d32487890c7b633e69088fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5285a4ac2d32487890c7b633e69088fe, State: Initialized, Role: FOLLOWER
I20260620 23:11:01.566475 13349 tablet_bootstrap.cc:492] T 72b6a646820a445c8eedcfcb62936d6a P a4532051e6c94424963f3862691fb2a4: No bootstrap required, opened a new log
I20260620 23:11:01.566471 13350 consensus_queue.cc:260] T 72b6a646820a445c8eedcfcb62936d6a P 5285a4ac2d32487890c7b633e69088fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } }
I20260620 23:11:01.566622 13349 ts_tablet_manager.cc:1397] T 72b6a646820a445c8eedcfcb62936d6a P a4532051e6c94424963f3862691fb2a4: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260620 23:11:01.567003 13351 raft_consensus.cc:348] T 72b6a646820a445c8eedcfcb62936d6a P 528ab34d28ba41f3b392a149fb596254 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } }
I20260620 23:11:01.567257 13351 raft_consensus.cc:374] T 72b6a646820a445c8eedcfcb62936d6a P 528ab34d28ba41f3b392a149fb596254 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:01.567337 13351 raft_consensus.cc:729] T 72b6a646820a445c8eedcfcb62936d6a P 528ab34d28ba41f3b392a149fb596254 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 528ab34d28ba41f3b392a149fb596254, State: Initialized, Role: FOLLOWER
I20260620 23:11:01.567344 13349 raft_consensus.cc:348] T 72b6a646820a445c8eedcfcb62936d6a P a4532051e6c94424963f3862691fb2a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } }
I20260620 23:11:01.567581 13349 raft_consensus.cc:374] T 72b6a646820a445c8eedcfcb62936d6a P a4532051e6c94424963f3862691fb2a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:01.567559 13351 consensus_queue.cc:260] T 72b6a646820a445c8eedcfcb62936d6a P 528ab34d28ba41f3b392a149fb596254 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } }
I20260620 23:11:01.567672 13349 raft_consensus.cc:729] T 72b6a646820a445c8eedcfcb62936d6a P a4532051e6c94424963f3862691fb2a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4532051e6c94424963f3862691fb2a4, State: Initialized, Role: FOLLOWER
I20260620 23:11:01.567939 13350 ts_tablet_manager.cc:1428] T 72b6a646820a445c8eedcfcb62936d6a P 5285a4ac2d32487890c7b633e69088fe: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260620 23:11:01.567925 13349 consensus_queue.cc:260] T 72b6a646820a445c8eedcfcb62936d6a P a4532051e6c94424963f3862691fb2a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } }
I20260620 23:11:01.568059 13351 ts_tablet_manager.cc:1428] T 72b6a646820a445c8eedcfcb62936d6a P 528ab34d28ba41f3b392a149fb596254: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:11:01.568439 13349 ts_tablet_manager.cc:1428] T 72b6a646820a445c8eedcfcb62936d6a P a4532051e6c94424963f3862691fb2a4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:11:01.809623 13355 raft_consensus.cc:482] T 72b6a646820a445c8eedcfcb62936d6a P 5285a4ac2d32487890c7b633e69088fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:11:01.809870 13355 raft_consensus.cc:504] T 72b6a646820a445c8eedcfcb62936d6a P 5285a4ac2d32487890c7b633e69088fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } }
I20260620 23:11:01.811499 13355 leader_election.cc:288] T 72b6a646820a445c8eedcfcb62936d6a P 5285a4ac2d32487890c7b633e69088fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4532051e6c94424963f3862691fb2a4 (127.10.191.193:40525), 528ab34d28ba41f3b392a149fb596254 (127.10.191.194:45137)
I20260620 23:11:01.819937 13366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:01.811890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:48125 (local address 127.10.191.193:40525)
0620 23:11:01.812854 (+ 964us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:01.812860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:01.812888 (+ 28us) server_negotiation.cc:407] Connection header received
0620 23:11:01.812948 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:01.812952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:01.813005 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:01.813082 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:11:01.813730 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:01.814806 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:01.817416 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:01.818668 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:01.818888 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:01.818951 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:01.819477 (+ 526us) server_negotiation.cc:299] Negotiation successful
0620 23:11:01.819610 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":817,"thread_start_us":674,"threads_started":1}
I20260620 23:11:01.819952 13358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:01.812636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:52817 (local address 127.10.191.194:45137)
0620 23:11:01.812822 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:01.812830 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:01.813275 (+ 445us) server_negotiation.cc:407] Connection header received
0620 23:11:01.813438 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:01.813441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:01.813495 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:01.813563 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:11:01.814068 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:01.815096 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:01.817484 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:01.818685 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:01.818838 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:01.818948 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:01.819477 (+ 529us) server_negotiation.cc:299] Negotiation successful
0620 23:11:01.819607 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:11:01.820113 13367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:01.812523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.194:45137 (local address 127.10.191.195:52817)
0620 23:11:01.813161 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:01.813177 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:01.813312 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:01.813603 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:01.813610 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:01.813641 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:11:01.813926 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:01.813934 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:01.815195 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:01.815199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:01.817264 (+ 2065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:01.817280 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:01.818958 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:01.818963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:01.819211 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:01.819340 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 23:11:01.819437 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 23:11:01.819630 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":124,"thread_start_us":309,"threads_started":1}
I20260620 23:11:01.820158 13365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:01.811699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:40525 (local address 127.10.191.195:48125)
0620 23:11:01.812181 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:01.812198 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:01.812317 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:01.813136 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:01.813144 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:01.813181 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:11:01.813493 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:01.813502 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:01.814963 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:01.814966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:01.817264 (+ 2298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:01.817280 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:01.818950 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:01.818957 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:01.819208 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:01.819340 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 23:11:01.819437 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 23:11:01.819630 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":156,"spinlock_wait_cycles":30464,"thread_start_us":231,"threads_started":1}
I20260620 23:11:01.820886 13043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72b6a646820a445c8eedcfcb62936d6a" candidate_uuid: "5285a4ac2d32487890c7b633e69088fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "528ab34d28ba41f3b392a149fb596254" is_pre_election: true
I20260620 23:11:01.820863 12904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72b6a646820a445c8eedcfcb62936d6a" candidate_uuid: "5285a4ac2d32487890c7b633e69088fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4532051e6c94424963f3862691fb2a4" is_pre_election: true
I20260620 23:11:01.821135 13043 raft_consensus.cc:2454] T 72b6a646820a445c8eedcfcb62936d6a P 528ab34d28ba41f3b392a149fb596254 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5285a4ac2d32487890c7b633e69088fe in term 0.
I20260620 23:11:01.821135 12904 raft_consensus.cc:2454] T 72b6a646820a445c8eedcfcb62936d6a P a4532051e6c94424963f3862691fb2a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5285a4ac2d32487890c7b633e69088fe in term 0.
I20260620 23:11:01.821611 13114 leader_election.cc:302] T 72b6a646820a445c8eedcfcb62936d6a P 5285a4ac2d32487890c7b633e69088fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5285a4ac2d32487890c7b633e69088fe, 528ab34d28ba41f3b392a149fb596254; no voters:
I20260620 23:11:01.821991 13355 raft_consensus.cc:2781] T 72b6a646820a445c8eedcfcb62936d6a P 5285a4ac2d32487890c7b633e69088fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:11:01.822117 13355 raft_consensus.cc:482] T 72b6a646820a445c8eedcfcb62936d6a P 5285a4ac2d32487890c7b633e69088fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:11:01.822213 13355 raft_consensus.cc:3037] T 72b6a646820a445c8eedcfcb62936d6a P 5285a4ac2d32487890c7b633e69088fe [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:01.824404 13355 raft_consensus.cc:504] T 72b6a646820a445c8eedcfcb62936d6a P 5285a4ac2d32487890c7b633e69088fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } }
I20260620 23:11:01.824980 13355 leader_election.cc:288] T 72b6a646820a445c8eedcfcb62936d6a P 5285a4ac2d32487890c7b633e69088fe [CANDIDATE]: Term 1 election: Requested vote from peers a4532051e6c94424963f3862691fb2a4 (127.10.191.193:40525), 528ab34d28ba41f3b392a149fb596254 (127.10.191.194:45137)
I20260620 23:11:01.825502 12904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72b6a646820a445c8eedcfcb62936d6a" candidate_uuid: "5285a4ac2d32487890c7b633e69088fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4532051e6c94424963f3862691fb2a4"
I20260620 23:11:01.825554 13043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72b6a646820a445c8eedcfcb62936d6a" candidate_uuid: "5285a4ac2d32487890c7b633e69088fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "528ab34d28ba41f3b392a149fb596254"
I20260620 23:11:01.825724 12904 raft_consensus.cc:3037] T 72b6a646820a445c8eedcfcb62936d6a P a4532051e6c94424963f3862691fb2a4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:01.825731 13043 raft_consensus.cc:3037] T 72b6a646820a445c8eedcfcb62936d6a P 528ab34d28ba41f3b392a149fb596254 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:01.828691 12904 raft_consensus.cc:2454] T 72b6a646820a445c8eedcfcb62936d6a P a4532051e6c94424963f3862691fb2a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5285a4ac2d32487890c7b633e69088fe in term 1.
I20260620 23:11:01.828760 13043 raft_consensus.cc:2454] T 72b6a646820a445c8eedcfcb62936d6a P 528ab34d28ba41f3b392a149fb596254 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5285a4ac2d32487890c7b633e69088fe in term 1.
I20260620 23:11:01.829078 13114 leader_election.cc:302] T 72b6a646820a445c8eedcfcb62936d6a P 5285a4ac2d32487890c7b633e69088fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5285a4ac2d32487890c7b633e69088fe, a4532051e6c94424963f3862691fb2a4; no voters:
I20260620 23:11:01.829350 13355 raft_consensus.cc:2781] T 72b6a646820a445c8eedcfcb62936d6a P 5285a4ac2d32487890c7b633e69088fe [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:11:01.829752 13355 raft_consensus.cc:686] T 72b6a646820a445c8eedcfcb62936d6a P 5285a4ac2d32487890c7b633e69088fe [term 1 LEADER]: Becoming Leader. State: Replica: 5285a4ac2d32487890c7b633e69088fe, State: Running, Role: LEADER
I20260620 23:11:01.830319 13355 consensus_queue.cc:237] T 72b6a646820a445c8eedcfcb62936d6a P 5285a4ac2d32487890c7b633e69088fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } }
I20260620 23:11:01.833693 12746 catalog_manager.cc:5495] T 72b6a646820a445c8eedcfcb62936d6a P 5285a4ac2d32487890c7b633e69088fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 5285a4ac2d32487890c7b633e69088fe (127.10.191.195). New cstate: current_term: 1 leader_uuid: "5285a4ac2d32487890c7b633e69088fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5285a4ac2d32487890c7b633e69088fe" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42307 } health_report { overall_health: HEALTHY } } }
I20260620 23:11:01.858055 13371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:01.852664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43318 (local address 127.10.191.195:42307)
0620 23:11:01.853171 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:01.853175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:01.853193 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:11:01.853265 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:01.853269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:01.853337 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:01.853472 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:11:01.853773 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:01.854587 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:01.855993 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:01.856365 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:01.856456 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:01.856850 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:11:01.857412 (+ 562us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:11:01.857525 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:01.857596 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 23:11:01.857655 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":306,"threads_started":1}
I20260620 23:11:01.872195 13366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:01.867234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:42335 (local address 127.10.191.193:40525)
0620 23:11:01.867426 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:01.867431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:01.867718 (+ 287us) server_negotiation.cc:407] Connection header received
0620 23:11:01.867848 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:01.867852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:01.867926 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:01.868006 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:11:01.868436 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:01.869141 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:01.870575 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:01.871111 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:01.871213 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:01.871621 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:11:01.871714 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:11:01.871793 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:01.871913 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 23:11:01.871970 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 23:11:01.872596 13374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:01.867023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:40525 (local address 127.10.191.195:42335)
0620 23:11:01.867626 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:01.867642 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:01.867747 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:01.868028 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:01.868031 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:01.868040 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:11:01.868308 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:01.868315 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:01.869273 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:01.869277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:01.870427 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:01.870439 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:01.871258 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:01.871263 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:01.871382 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:01.871428 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:11:01.871800 (+ 372us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:11:01.871809 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 23:11:01.871886 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 23:11:01.871978 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":294,"threads_started":1}
I20260620 23:11:01.880132 13043 raft_consensus.cc:1261] T 72b6a646820a445c8eedcfcb62936d6a P 528ab34d28ba41f3b392a149fb596254 [term 1 FOLLOWER]: Refusing update from remote peer 5285a4ac2d32487890c7b633e69088fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:11:01.880208 12904 raft_consensus.cc:1261] T 72b6a646820a445c8eedcfcb62936d6a P a4532051e6c94424963f3862691fb2a4 [term 1 FOLLOWER]: Refusing update from remote peer 5285a4ac2d32487890c7b633e69088fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:11:01.880786 13368 consensus_queue.cc:1035] T 72b6a646820a445c8eedcfcb62936d6a P 5285a4ac2d32487890c7b633e69088fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "528ab34d28ba41f3b392a149fb596254" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:11:01.881191 13355 consensus_queue.cc:1035] T 72b6a646820a445c8eedcfcb62936d6a P 5285a4ac2d32487890c7b633e69088fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4532051e6c94424963f3862691fb2a4" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:11:01.909888 13371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:01.904171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:46753 (local address 127.10.191.195:42307)
0620 23:11:01.904314 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:01.904318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:01.904814 (+ 496us) server_negotiation.cc:407] Connection header received
0620 23:11:01.904996 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:01.905001 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:01.905071 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:01.905151 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:11:01.905597 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:01.906623 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:01.908295 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:01.908701 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:01.908857 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:01.909260 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:11:01.909348 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:11:01.909436 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:01.909543 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 23:11:01.909613 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 23:11:01.910068 13379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:01.904192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.195:42307 (local address 127.10.191.193:46753)
0620 23:11:01.904717 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:01.904732 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:01.904883 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:01.905177 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:01.905181 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:01.905196 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:11:01.905456 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:01.905464 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:01.906743 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:01.906786 (+ 43us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:01.908166 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:01.908178 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:01.908870 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:01.908876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:01.909051 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:01.909085 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:11:01.909438 (+ 353us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:11:01.909451 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 23:11:01.909526 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 23:11:01.909649 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":292,"threads_started":1}
W20260620 23:11:01.911041 13320 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 23:11:03.500710 12747 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:11:03.500967 12747 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:11:03.504666 12747 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 69ad19c2711d4c08b9b99bd0fe396704: Sending DeleteTablet for 3 replicas of tablet 72b6a646820a445c8eedcfcb62936d6a
I20260620 23:11:03.505460 13162 tablet_service.cc:1514] Processing DeleteTablet for tablet 72b6a646820a445c8eedcfcb62936d6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:11:03 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:43302
I20260620 23:11:03.505508 12883 tablet_service.cc:1514] Processing DeleteTablet for tablet 72b6a646820a445c8eedcfcb62936d6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:11:03 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:45896
I20260620 23:11:03.505532 13022 tablet_service.cc:1514] Processing DeleteTablet for tablet 72b6a646820a445c8eedcfcb62936d6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:11:03 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:51198
I20260620 23:11:03.506054 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 12818
I20260620 23:11:03.509189 13402 tablet_replica.cc:330] T 72b6a646820a445c8eedcfcb62936d6a P 5285a4ac2d32487890c7b633e69088fe: stopping tablet replica
I20260620 23:11:03.509531 13402 raft_consensus.cc:2229] T 72b6a646820a445c8eedcfcb62936d6a P 5285a4ac2d32487890c7b633e69088fe [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:11:03.510241 13402 raft_consensus.cc:2258] T 72b6a646820a445c8eedcfcb62936d6a P 5285a4ac2d32487890c7b633e69088fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:11:03.512035 13402 ts_tablet_manager.cc:1905] T 72b6a646820a445c8eedcfcb62936d6a P 5285a4ac2d32487890c7b633e69088fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:11:03.520376 13402 ts_tablet_manager.cc:1918] T 72b6a646820a445c8eedcfcb62936d6a P 5285a4ac2d32487890c7b633e69088fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 23:11:03.520597 13402 log.cc:1196] T 72b6a646820a445c8eedcfcb62936d6a P 5285a4ac2d32487890c7b633e69088fe: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-2/wal/wals/72b6a646820a445c8eedcfcb62936d6a
I20260620 23:11:03.520865 13404 tablet_replica.cc:330] T 72b6a646820a445c8eedcfcb62936d6a P a4532051e6c94424963f3862691fb2a4: stopping tablet replica
I20260620 23:11:03.521026 13403 tablet_replica.cc:330] T 72b6a646820a445c8eedcfcb62936d6a P 528ab34d28ba41f3b392a149fb596254: stopping tablet replica
I20260620 23:11:03.521179 13402 ts_tablet_manager.cc:1939] T 72b6a646820a445c8eedcfcb62936d6a P 5285a4ac2d32487890c7b633e69088fe: Deleting consensus metadata
I20260620 23:11:03.521221 13403 raft_consensus.cc:2229] T 72b6a646820a445c8eedcfcb62936d6a P 528ab34d28ba41f3b392a149fb596254 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:11:03.521303 13404 raft_consensus.cc:2229] T 72b6a646820a445c8eedcfcb62936d6a P a4532051e6c94424963f3862691fb2a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:11:03.521469 13403 raft_consensus.cc:2258] T 72b6a646820a445c8eedcfcb62936d6a P 528ab34d28ba41f3b392a149fb596254 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:11:03.522032 13404 raft_consensus.cc:2258] T 72b6a646820a445c8eedcfcb62936d6a P a4532051e6c94424963f3862691fb2a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:11:03.522249 12733 catalog_manager.cc:4841] TS 5285a4ac2d32487890c7b633e69088fe (127.10.191.195:42307): tablet 72b6a646820a445c8eedcfcb62936d6a (table test-table [id=6274c724193b4c49a2dc9873b4c59e08]) successfully deleted
I20260620 23:11:03.522362 13403 ts_tablet_manager.cc:1905] T 72b6a646820a445c8eedcfcb62936d6a P 528ab34d28ba41f3b392a149fb596254: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:11:03.523233 13404 ts_tablet_manager.cc:1905] T 72b6a646820a445c8eedcfcb62936d6a P a4532051e6c94424963f3862691fb2a4: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 23:11:03 dist-test-slave-jkbq krb5kdc[12702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997061, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.191.193@KRBTEST.COM
I20260620 23:11:03.528962 13403 ts_tablet_manager.cc:1918] T 72b6a646820a445c8eedcfcb62936d6a P 528ab34d28ba41f3b392a149fb596254: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 23:11:03.529085 13403 log.cc:1196] T 72b6a646820a445c8eedcfcb62936d6a P 528ab34d28ba41f3b392a149fb596254: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-1/wal/wals/72b6a646820a445c8eedcfcb62936d6a
I20260620 23:11:03.529532 13403 ts_tablet_manager.cc:1939] T 72b6a646820a445c8eedcfcb62936d6a P 528ab34d28ba41f3b392a149fb596254: Deleting consensus metadata
I20260620 23:11:03.529649 13404 ts_tablet_manager.cc:1918] T 72b6a646820a445c8eedcfcb62936d6a P a4532051e6c94424963f3862691fb2a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 23:11:03.529757 13404 log.cc:1196] T 72b6a646820a445c8eedcfcb62936d6a P a4532051e6c94424963f3862691fb2a4: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781997042893763-11007-0/minicluster-data/ts-0/wal/wals/72b6a646820a445c8eedcfcb62936d6a
I20260620 23:11:03.530241 13404 ts_tablet_manager.cc:1939] T 72b6a646820a445c8eedcfcb62936d6a P a4532051e6c94424963f3862691fb2a4: Deleting consensus metadata
I20260620 23:11:03.530709 12732 catalog_manager.cc:4841] TS 528ab34d28ba41f3b392a149fb596254 (127.10.191.194:45137): tablet 72b6a646820a445c8eedcfcb62936d6a (table test-table [id=6274c724193b4c49a2dc9873b4c59e08]) successfully deleted
I20260620 23:11:03.531299 12732 catalog_manager.cc:4841] TS a4532051e6c94424963f3862691fb2a4 (127.10.191.193:40525): tablet 72b6a646820a445c8eedcfcb62936d6a (table test-table [id=6274c724193b4c49a2dc9873b4c59e08]) successfully deleted
I20260620 23:11:03.535037 13406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:03.507305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40812 (local address 127.10.191.193:40525)
0620 23:11:03.508819 (+ 1514us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:03.508828 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:03.518919 (+ 10091us) server_negotiation.cc:407] Connection header received
0620 23:11:03.519363 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:03.519369 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:03.519447 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:03.519520 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:03.521505 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:03.522613 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:03.523998 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:03.524452 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:03.524603 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:03.528827 (+ 4224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:03.528841 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:03.528857 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:03.528894 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:03.532390 (+ 3496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:03.533006 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:03.533013 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:03.533016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:03.533142 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:03.533522 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:03.533526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:03.533528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:03.533880 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:03.534165 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:03.534440 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 23:11:03.534646 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1364,"thread_start_us":1253,"threads_started":1}
W20260620 23:11:03.535641 12949 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40812
W20260620 23:11:03.536219 11007 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 12818: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:11:03.536441 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 12818
I20260620 23:11:03.561138 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 12958
Jun 20 23:11:03 dist-test-slave-jkbq krb5kdc[12702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997061, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.191.194@KRBTEST.COM
I20260620 23:11:03.576153 13407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:03.561849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36018 (local address 127.10.191.194:45137)
0620 23:11:03.562416 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:03.562421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:03.562439 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:11:03.562495 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:03.562499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:03.562563 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:03.562659 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:03.563860 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:03.564607 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:03.565798 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:03.566204 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:03.566309 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:03.570590 (+ 4281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:03.570605 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:03.570620 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:03.570653 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:03.573505 (+ 2852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:03.574218 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:03.574227 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:03.574230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:03.574364 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:03.574647 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:03.574650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:03.574652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:03.575086 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:03.575309 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:03.575566 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 23:11:03.575779 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":292,"threads_started":1}
W20260620 23:11:03.576785 13088 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36018
W20260620 23:11:03.577333 11007 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 12958: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:11:03.577461 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 12958
I20260620 23:11:03.598989 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 13097
Jun 20 23:11:03 dist-test-slave-jkbq krb5kdc[12702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997061, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.191.195@KRBTEST.COM
I20260620 23:11:03.615720 13408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:03.599747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42410 (local address 127.10.191.195:42307)
0620 23:11:03.600246 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:03.600252 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:03.600272 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:11:03.600333 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:03.600355 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:03.600428 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:03.600533 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:03.601784 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:03.602950 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:03.604120 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:03.604776 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:03.604900 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:03.609073 (+ 4173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:03.609090 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:03.609111 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:03.609153 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:03.612930 (+ 3777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:03.613580 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:03.613589 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:03.613593 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:03.613747 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:03.614083 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:03.614088 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:03.614091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:03.614506 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:03.614786 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:03.615126 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 23:11:03.615333 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":229,"threads_started":1}
W20260620 23:11:03.616401 13227 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42410
W20260620 23:11:03.617040 11007 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 13097: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:11:03.617218 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 13097
I20260620 23:11:03.638609 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 12715
I20260620 23:11:03.639889 12787 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:11:03.753306 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 12715
2026-06-20T23:11:03Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8489 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:11:03 dist-test-slave-jkbq krb5kdc[13412](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:11:03 dist-test-slave-jkbq krb5kdc[13412](info): set up 2 sockets
Jun 20 23:11:03 dist-test-slave-jkbq krb5kdc[13412](info): commencing operation
krb5kdc: starting...
W20260620 23:11:04.842942 11007 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Jun 20 23:11:04 dist-test-slave-jkbq krb5kdc[13412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:11:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:11:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.191.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
I20260620 23:11:05.024102 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:35855
--webserver_interface=127.10.191.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:43545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:35855
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:11:05.336431 13425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:05.336755 13425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:05.336939 13425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:05.343876 13425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:11:05.344018 13425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:05.344105 13425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:05.344185 13425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:11:05.344255 13425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:11:05.353756 13425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:43545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:35855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:35855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:05.355141 13425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:05.356492 13425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:05.368480 13430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:05.369024 13431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:05.370047 13433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:05.370610 13425 server_base.cc:1034] running on GCE node
I20260620 23:11:05.371467 13425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:05.372814 13425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:05.374023 13425 hybrid_clock.cc:648] HybridClock initialized: now 1781997065373980 us; error 29 us; skew 500 ppm
I20260620 23:11:05.374384 13425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:11:05 dist-test-slave-jkbq krb5kdc[13412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997065, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:05.377504 13425 init.cc:377] Logged in from keytab as kudu/127.10.191.254@KRBTEST.COM (short username kudu)
I20260620 23:11:05.379021 13425 webserver.cc:460] Webserver started at http://127.10.191.254:43409/ using document root <none> and password file <none>
I20260620 23:11:05.379720 13425 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:05.379905 13425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:05.380220 13425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:11:05.383183 13425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "5e62ed8662104a02a402078214fccd86"
format_stamp: "Formatted at 2026-06-20 23:11:05 on dist-test-slave-jkbq"
server_key: "12cc8b5708163565aaf49244da372b51"
server_key_iv: "e420e84e5ff9c924e59f1ced566e65c7"
server_key_version: "encryptionkey@0"
I20260620 23:11:05.383980 13425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "5e62ed8662104a02a402078214fccd86"
format_stamp: "Formatted at 2026-06-20 23:11:05 on dist-test-slave-jkbq"
server_key: "12cc8b5708163565aaf49244da372b51"
server_key_iv: "e420e84e5ff9c924e59f1ced566e65c7"
server_key_version: "encryptionkey@0"
I20260620 23:11:05.389026 13425 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:11:05.392424 13440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:05.393702 13425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:11:05.393894 13425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "5e62ed8662104a02a402078214fccd86"
format_stamp: "Formatted at 2026-06-20 23:11:05 on dist-test-slave-jkbq"
server_key: "12cc8b5708163565aaf49244da372b51"
server_key_iv: "e420e84e5ff9c924e59f1ced566e65c7"
server_key_version: "encryptionkey@0"
I20260620 23:11:05.394081 13425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:05.443863 13425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:05.447718 13425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:05.448005 13425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:05.466356 13425 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:35855
I20260620 23:11:05.466377 13491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:35855 every 8 connection(s)
I20260620 23:11:05.467741 13425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:11:05.470604 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 13425
I20260620 23:11:05.470831 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:11:05.471408 13492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:11:05.471508 11007 external_mini_cluster.cc:1428] Setting key 38e6a17d223c1f4f80deb86ef01d017b
I20260620 23:11:05.485401 13492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e62ed8662104a02a402078214fccd86: Bootstrap starting.
Jun 20 23:11:05 dist-test-slave-jkbq krb5kdc[13412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:11:05.489125 13492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e62ed8662104a02a402078214fccd86: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:05.490398 13492 log.cc:824] T 00000000000000000000000000000000 P 5e62ed8662104a02a402078214fccd86: Log is configured to *not* fsync() on all Append() calls
I20260620 23:11:05.493655 13492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e62ed8662104a02a402078214fccd86: No bootstrap required, opened a new log
I20260620 23:11:05.494807 13495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:05.473373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43964 (local address 127.10.191.254:35855)
0620 23:11:05.474169 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:05.474184 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:05.474223 (+ 39us) server_negotiation.cc:407] Connection header received
0620 23:11:05.475888 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:05.475908 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:05.476279 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:05.476590 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:05.478314 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:05.479626 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:05.481333 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:05.482007 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:05.482147 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:05.486609 (+ 4462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:05.486619 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:05.486638 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:05.486688 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:05.490502 (+ 3814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:05.491382 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:05.491388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:05.491394 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:05.491546 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:05.491943 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:05.491947 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:05.491950 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:05.492548 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:05.492827 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:05.493150 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 23:11:05.493532 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":400,"threads_started":1}
I20260620 23:11:05.501816 13492 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e62ed8662104a02a402078214fccd86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e62ed8662104a02a402078214fccd86" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 35855 } }
I20260620 23:11:05.502135 13492 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e62ed8662104a02a402078214fccd86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:05.502205 13492 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e62ed8662104a02a402078214fccd86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e62ed8662104a02a402078214fccd86, State: Initialized, Role: FOLLOWER
I20260620 23:11:05.502678 13492 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e62ed8662104a02a402078214fccd86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e62ed8662104a02a402078214fccd86" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 35855 } }
I20260620 23:11:05.502943 13492 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e62ed8662104a02a402078214fccd86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:11:05.503048 13492 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e62ed8662104a02a402078214fccd86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:11:05.503187 13492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e62ed8662104a02a402078214fccd86 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:05.505417 13492 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e62ed8662104a02a402078214fccd86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e62ed8662104a02a402078214fccd86" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 35855 } }
I20260620 23:11:05.505869 13492 leader_election.cc:302] T 00000000000000000000000000000000 P 5e62ed8662104a02a402078214fccd86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e62ed8662104a02a402078214fccd86; no voters:
I20260620 23:11:05.506419 13492 leader_election.cc:288] T 00000000000000000000000000000000 P 5e62ed8662104a02a402078214fccd86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:11:05.506505 13497 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e62ed8662104a02a402078214fccd86 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:11:05.507771 13497 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e62ed8662104a02a402078214fccd86 [term 1 LEADER]: Becoming Leader. State: Replica: 5e62ed8662104a02a402078214fccd86, State: Running, Role: LEADER
I20260620 23:11:05.508380 13497 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e62ed8662104a02a402078214fccd86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e62ed8662104a02a402078214fccd86" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 35855 } }
I20260620 23:11:05.508565 13492 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e62ed8662104a02a402078214fccd86 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:11:05.511662 13499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e62ed8662104a02a402078214fccd86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e62ed8662104a02a402078214fccd86. Latest consensus state: current_term: 1 leader_uuid: "5e62ed8662104a02a402078214fccd86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e62ed8662104a02a402078214fccd86" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 35855 } } }
I20260620 23:11:05.512020 13499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e62ed8662104a02a402078214fccd86 [sys.catalog]: This master's current role is: LEADER
I20260620 23:11:05.511765 13498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e62ed8662104a02a402078214fccd86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e62ed8662104a02a402078214fccd86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e62ed8662104a02a402078214fccd86" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 35855 } } }
I20260620 23:11:05.512164 13498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e62ed8662104a02a402078214fccd86 [sys.catalog]: This master's current role is: LEADER
I20260620 23:11:05.513806 13504 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:11:05.518765 13504 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:11:05.528721 13504 catalog_manager.cc:1347] Generated new cluster ID: fe31a12e5e01416db28fe281394e68f2
I20260620 23:11:05.528844 13504 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:11:05.545842 13504 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:11:05.547128 13504 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:11:05.557930 13504 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e62ed8662104a02a402078214fccd86: Generated new TSK 0
I20260620 23:11:05.558689 13504 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.191.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab.
Entry for principal kudu/127.10.191.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab.
I20260620 23:11:05.619161 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.193:0
--local_ip_for_outbound_sockets=127.10.191.193
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:35855
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:43545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:11:05.949064 13518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:05.949316 13518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:05.949405 13518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:05.956272 13518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:05.956363 13518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:05.956467 13518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.193
I20260620 23:11:05.966475 13518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:43545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:35855
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.193
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:05.967657 13518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:05.968880 13518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:05.981174 13523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:05.981153 13526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:05.981282 13524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:05.982219 13518 server_base.cc:1034] running on GCE node
I20260620 23:11:05.982791 13518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:05.983687 13518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:05.984897 13518 hybrid_clock.cc:648] HybridClock initialized: now 1781997065984832 us; error 51 us; skew 500 ppm
I20260620 23:11:05.985159 13518 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:11:05 dist-test-slave-jkbq krb5kdc[13412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997065, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:05.988294 13518 init.cc:377] Logged in from keytab as kudu/127.10.191.193@KRBTEST.COM (short username kudu)
I20260620 23:11:05.989753 13518 webserver.cc:460] Webserver started at http://127.10.191.193:37271/ using document root <none> and password file <none>
I20260620 23:11:05.990386 13518 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:05.990480 13518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:05.990722 13518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:11:05.993633 13518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-0/data/instance:
uuid: "6b7bb23ed44d41bb99bd31c0f94ce655"
format_stamp: "Formatted at 2026-06-20 23:11:05 on dist-test-slave-jkbq"
server_key: "c86e069d30c9621006db5f8f376e7513"
server_key_iv: "078969768148a95ec36e44b81af36d82"
server_key_version: "encryptionkey@0"
I20260620 23:11:05.994401 13518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance:
uuid: "6b7bb23ed44d41bb99bd31c0f94ce655"
format_stamp: "Formatted at 2026-06-20 23:11:05 on dist-test-slave-jkbq"
server_key: "c86e069d30c9621006db5f8f376e7513"
server_key_iv: "078969768148a95ec36e44b81af36d82"
server_key_version: "encryptionkey@0"
I20260620 23:11:05.999573 13518 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 23:11:06.003052 13533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:06.004388 13518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:11:06.004570 13518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-0/wal
uuid: "6b7bb23ed44d41bb99bd31c0f94ce655"
format_stamp: "Formatted at 2026-06-20 23:11:05 on dist-test-slave-jkbq"
server_key: "c86e069d30c9621006db5f8f376e7513"
server_key_iv: "078969768148a95ec36e44b81af36d82"
server_key_version: "encryptionkey@0"
I20260620 23:11:06.004757 13518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:06.031111 13518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:06.035388 13518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:06.035667 13518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:06.036581 13518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:11:06.037932 13518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:11:06.038028 13518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:06.038167 13518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:11:06.038234 13518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:06.070284 13518 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.193:34689
I20260620 23:11:06.070340 13645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.193:34689 every 8 connection(s)
I20260620 23:11:06.071620 13518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
I20260620 23:11:06.078809 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 13518
I20260620 23:11:06.079057 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance
I20260620 23:11:06.079674 11007 external_mini_cluster.cc:1428] Setting key e2442cb71ae3483a2cf175a51d445f39
Jun 20 23:11:06 dist-test-slave-jkbq krb5kdc[13412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997065, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:11:06.091647 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:06.074229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:57457 (local address 127.10.191.254:35855)
0620 23:11:06.074643 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:06.074647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:06.075721 (+ 1074us) server_negotiation.cc:407] Connection header received
0620 23:11:06.077579 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:06.077585 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:06.077673 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:06.077788 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:06.079348 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:06.080124 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:06.081937 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:06.082337 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:06.082465 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:06.086070 (+ 3605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:06.086075 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:06.086078 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:06.086107 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:06.088594 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:06.089342 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:06.089347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:06.089351 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:06.089470 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:06.089899 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:06.089908 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:06.089910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:06.090207 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:06.090395 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:06.091067 (+ 672us) server_negotiation.cc:299] Negotiation successful
0620 23:11:06.091287 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":213,"threads_started":1}
I20260620 23:11:06.092547 13649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:06.074784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:35855 (local address 127.10.191.193:57457)
0620 23:11:06.075543 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:06.075591 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:06.077312 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:06.078026 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:06.078033 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:06.078509 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:06.079173 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:06.079197 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:06.080273 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:06.080278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:06.081791 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:06.081808 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:06.082559 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:06.082566 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:06.082816 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:06.083543 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:06.083564 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:06.085877 (+ 2313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:06.088768 (+ 2891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:06.088783 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:06.088799 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:06.089220 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:06.089570 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:06.089576 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:06.089580 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:06.089752 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:06.090342 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:06.090350 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:06.090679 (+ 329us) client_negotiation.cc:769] Sending connection context
0620 23:11:06.090975 (+ 296us) client_negotiation.cc:240] Negotiation successful
0620 23:11:06.091472 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":281,"threads_started":1}
I20260620 23:11:06.094079 13646 heartbeater.cc:347] Connected to a master server at 127.10.191.254:35855
I20260620 23:11:06.094370 13646 heartbeater.cc:464] Registering TS with master...
I20260620 23:11:06.095083 13646 heartbeater.cc:511] Master 127.10.191.254:35855 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.191.194@KRBTEST.COM; defaulting to no policy
I20260620 23:11:06.097154 13457 ts_manager.cc:194] Registered new tserver with Master: 6b7bb23ed44d41bb99bd31c0f94ce655 (127.10.191.193:34689)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.194@KRBTEST.COM" created.
I20260620 23:11:06.099054 13457 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.193@KRBTEST.COM'} at 127.10.191.193:57457
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab.
Entry for principal kudu/127.10.191.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab.
I20260620 23:11:06.121512 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.194:0
--local_ip_for_outbound_sockets=127.10.191.194
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:35855
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:43545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:11:06.440258 13652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:06.440523 13652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:06.440672 13652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:06.447829 13652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:06.447935 13652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:06.448089 13652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.194
I20260620 23:11:06.457610 13652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:43545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:35855
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.194
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:06.458961 13652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:06.460253 13652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:06.471992 13657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:06.473328 13658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:06.474296 13660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:06.474612 13652 server_base.cc:1034] running on GCE node
I20260620 23:11:06.475265 13652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:06.476267 13652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:06.477490 13652 hybrid_clock.cc:648] HybridClock initialized: now 1781997066477439 us; error 47 us; skew 500 ppm
I20260620 23:11:06.477803 13652 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:11:06 dist-test-slave-jkbq krb5kdc[13412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997066, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:06.484905 13652 init.cc:377] Logged in from keytab as kudu/127.10.191.194@KRBTEST.COM (short username kudu)
I20260620 23:11:06.486465 13652 webserver.cc:460] Webserver started at http://127.10.191.194:37943/ using document root <none> and password file <none>
I20260620 23:11:06.487232 13652 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:06.487360 13652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:06.487659 13652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:11:06.490439 13652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-1/data/instance:
uuid: "56118544c2244764af7fd36c24407cd5"
format_stamp: "Formatted at 2026-06-20 23:11:06 on dist-test-slave-jkbq"
server_key: "291bc1071feb1b1dd1ed00915f06b963"
server_key_iv: "b5ff9694cf35379ad92d0cc64bee9682"
server_key_version: "encryptionkey@0"
I20260620 23:11:06.491221 13652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance:
uuid: "56118544c2244764af7fd36c24407cd5"
format_stamp: "Formatted at 2026-06-20 23:11:06 on dist-test-slave-jkbq"
server_key: "291bc1071feb1b1dd1ed00915f06b963"
server_key_iv: "b5ff9694cf35379ad92d0cc64bee9682"
server_key_version: "encryptionkey@0"
I20260620 23:11:06.496083 13652 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 23:11:06.499655 13667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:06.500876 13652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:11:06.501039 13652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-1/wal
uuid: "56118544c2244764af7fd36c24407cd5"
format_stamp: "Formatted at 2026-06-20 23:11:06 on dist-test-slave-jkbq"
server_key: "291bc1071feb1b1dd1ed00915f06b963"
server_key_iv: "b5ff9694cf35379ad92d0cc64bee9682"
server_key_version: "encryptionkey@0"
I20260620 23:11:06.501230 13652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:06.514434 13652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:06.518503 13652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:06.518783 13652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:06.519650 13652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:11:06.520886 13652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:11:06.520984 13652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:06.521126 13652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:11:06.521194 13652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:06.552426 13652 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.194:38289
I20260620 23:11:06.552464 13779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.194:38289 every 8 connection(s)
I20260620 23:11:06.553715 13652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
I20260620 23:11:06.557500 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 13652
I20260620 23:11:06.557744 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance
I20260620 23:11:06.558202 11007 external_mini_cluster.cc:1428] Setting key 0331eb2d35c13137fbc72abb752c9349
Jun 20 23:11:06 dist-test-slave-jkbq krb5kdc[13412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997066, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:11:06.575516 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:06.556251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:50575 (local address 127.10.191.254:35855)
0620 23:11:06.556455 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:06.556461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:06.557681 (+ 1220us) server_negotiation.cc:407] Connection header received
0620 23:11:06.559763 (+ 2082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:06.559768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:06.559843 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:06.559958 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:06.561442 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:06.562133 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:06.563811 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:06.564331 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:06.564431 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:06.568562 (+ 4131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:06.568571 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:06.568579 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:06.568625 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:06.572214 (+ 3589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:06.572902 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:06.572911 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:06.572917 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:06.573080 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:06.573584 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:06.573591 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:06.573597 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:06.573992 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:06.574151 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:06.574851 (+ 700us) server_negotiation.cc:299] Negotiation successful
0620 23:11:06.575137 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 23:11:06.576210 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:06.556681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:35855 (local address 127.10.191.194:50575)
0620 23:11:06.557506 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:06.557557 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:06.559510 (+ 1953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:06.560206 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:06.560215 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:06.560726 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:06.561272 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:06.561291 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:06.562282 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:06.562287 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:06.563616 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:06.563629 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:06.564549 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:06.564557 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:06.564915 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:06.565738 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:06.565772 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:06.568348 (+ 2576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:06.572347 (+ 3999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:06.572353 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:06.572367 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:06.572765 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:06.573252 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:06.573257 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:06.573261 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:06.573455 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:06.574139 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:06.574145 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:06.574462 (+ 317us) client_negotiation.cc:769] Sending connection context
0620 23:11:06.574685 (+ 223us) client_negotiation.cc:240] Negotiation successful
0620 23:11:06.575133 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":261,"threads_started":1}
WARNING: no policy specified for kudu/127.10.191.195@KRBTEST.COM; defaulting to no policy
I20260620 23:11:06.577728 13780 heartbeater.cc:347] Connected to a master server at 127.10.191.254:35855
I20260620 23:11:06.578034 13780 heartbeater.cc:464] Registering TS with master...
I20260620 23:11:06.578848 13780 heartbeater.cc:511] Master 127.10.191.254:35855 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.195@KRBTEST.COM" created.
I20260620 23:11:06.580329 13457 ts_manager.cc:194] Registered new tserver with Master: 56118544c2244764af7fd36c24407cd5 (127.10.191.194:38289)
I20260620 23:11:06.581123 13457 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.194@KRBTEST.COM'} at 127.10.191.194:50575
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab.
Entry for principal kudu/127.10.191.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab.
I20260620 23:11:06.602005 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.195:0
--local_ip_for_outbound_sockets=127.10.191.195
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:35855
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:43545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:11:06.920137 13785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:06.920413 13785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:06.920506 13785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:06.927623 13785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:06.927736 13785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:06.927856 13785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.195
I20260620 23:11:06.937820 13785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:43545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:35855
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.195
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:06.939064 13785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:06.940338 13785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:06.954838 13793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:06.955045 13790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:06.955003 13791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:06.955821 13785 server_base.cc:1034] running on GCE node
I20260620 23:11:06.956367 13785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:06.957198 13785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:06.958431 13785 hybrid_clock.cc:648] HybridClock initialized: now 1781997066958372 us; error 71 us; skew 500 ppm
I20260620 23:11:06.958739 13785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:11:06 dist-test-slave-jkbq krb5kdc[13412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997066, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:06.961804 13785 init.cc:377] Logged in from keytab as kudu/127.10.191.195@KRBTEST.COM (short username kudu)
I20260620 23:11:06.963378 13785 webserver.cc:460] Webserver started at http://127.10.191.195:38225/ using document root <none> and password file <none>
I20260620 23:11:06.964048 13785 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:06.964186 13785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:06.964488 13785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:11:06.967485 13785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-2/data/instance:
uuid: "01664cf0f5b546e09b709dd4834b995a"
format_stamp: "Formatted at 2026-06-20 23:11:06 on dist-test-slave-jkbq"
server_key: "8f56d6ae4e0a5a3e10f173e4cff9324f"
server_key_iv: "b5761ccd5086ee7813883dd91d7621d5"
server_key_version: "encryptionkey@0"
I20260620 23:11:06.968246 13785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance:
uuid: "01664cf0f5b546e09b709dd4834b995a"
format_stamp: "Formatted at 2026-06-20 23:11:06 on dist-test-slave-jkbq"
server_key: "8f56d6ae4e0a5a3e10f173e4cff9324f"
server_key_iv: "b5761ccd5086ee7813883dd91d7621d5"
server_key_version: "encryptionkey@0"
I20260620 23:11:06.973245 13785 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 23:11:06.976636 13800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:06.978050 13785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:11:06.978169 13785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-2/wal
uuid: "01664cf0f5b546e09b709dd4834b995a"
format_stamp: "Formatted at 2026-06-20 23:11:06 on dist-test-slave-jkbq"
server_key: "8f56d6ae4e0a5a3e10f173e4cff9324f"
server_key_iv: "b5761ccd5086ee7813883dd91d7621d5"
server_key_version: "encryptionkey@0"
I20260620 23:11:06.978328 13785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:07.006690 13785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:07.011258 13785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:07.011497 13785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:07.012390 13785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:11:07.013738 13785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:11:07.013854 13785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:07.013993 13785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:11:07.014039 13785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:07.046568 13785 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.195:40673
I20260620 23:11:07.046619 13912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.195:40673 every 8 connection(s)
I20260620 23:11:07.047881 13785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
I20260620 23:11:07.049216 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 13785
I20260620 23:11:07.049397 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance
I20260620 23:11:07.049827 11007 external_mini_cluster.cc:1428] Setting key a57cfc84642070143adb59cee5d31865
Jun 20 23:11:07 dist-test-slave-jkbq krb5kdc[13412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997066, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:11:07.068457 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:07.050821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:33881 (local address 127.10.191.254:35855)
0620 23:11:07.051037 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:07.051044 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:07.052346 (+ 1302us) server_negotiation.cc:407] Connection header received
0620 23:11:07.054235 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:07.054240 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:07.054302 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:07.054390 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:07.055894 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:07.056668 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:07.058269 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:07.058694 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:07.058823 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:07.062452 (+ 3629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:07.062457 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:07.062460 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:07.062494 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:07.065717 (+ 3223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:07.066376 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:07.066380 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:07.066382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:07.066487 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:07.066836 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:07.066839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:07.066841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:07.067142 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:07.067247 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:07.067913 (+ 666us) server_negotiation.cc:299] Negotiation successful
0620 23:11:07.068133 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 23:11:07.069401 13915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:07.051356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:35855 (local address 127.10.191.195:33881)
0620 23:11:07.052159 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:07.052230 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:07.053984 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:07.054581 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:07.054588 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:07.055157 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:07.055732 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:07.055750 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:07.056851 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:07.056855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:07.058124 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:07.058136 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:07.058992 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:07.058999 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:07.059205 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:07.059905 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:07.059928 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:07.062241 (+ 2313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:07.065833 (+ 3592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:07.065840 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:07.065854 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:07.066278 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:07.066568 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:07.066572 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:07.066574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:07.066736 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:07.067251 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:07.067259 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:07.067574 (+ 315us) client_negotiation.cc:769] Sending connection context
0620 23:11:07.067816 (+ 242us) client_negotiation.cc:240] Negotiation successful
0620 23:11:07.068189 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":306,"threads_started":1}
I20260620 23:11:07.070700 13913 heartbeater.cc:347] Connected to a master server at 127.10.191.254:35855
I20260620 23:11:07.071034 13913 heartbeater.cc:464] Registering TS with master...
I20260620 23:11:07.071700 13913 heartbeater.cc:511] Master 127.10.191.254:35855 requested a full tablet report, sending...
I20260620 23:11:07.073118 13457 ts_manager.cc:194] Registered new tserver with Master: 01664cf0f5b546e09b709dd4834b995a (127.10.191.195:40673)
I20260620 23:11:07.073937 13457 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.195@KRBTEST.COM'} at 127.10.191.195:33881
I20260620 23:11:07.077409 11007 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:11:07.102497 13646 heartbeater.cc:503] Master 127.10.191.254:35855 was elected leader, sending a full tablet report...
W20260620 23:11:07.105635 13648 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:11:07.103249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43972 (local address 127.10.191.254:35855)
0620 23:11:07.103435 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:07.103440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:07.103950 (+ 510us) server_negotiation.cc:407] Connection header received
0620 23:11:07.104091 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:07.104096 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:07.104168 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:07.104288 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:07.105452 (+ 1164us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43972: BlockingRecv error: recv got EOF from 127.0.0.1:43972 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 23:11:07.109185 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 13518
W20260620 23:11:07.111925 11007 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 13518: Not authorized: Client connection negotiation failed: client connection to 127.10.191.193:34689: server requires authentication, but client does not have Kerberos credentials available
I20260620 23:11:07.112107 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 13518
I20260620 23:11:07.128041 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 13652
W20260620 23:11:07.130534 11007 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 13652: Not authorized: Client connection negotiation failed: client connection to 127.10.191.194:38289: server requires authentication, but client does not have Kerberos credentials available
I20260620 23:11:07.130688 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 13652
I20260620 23:11:07.145481 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 13785
W20260620 23:11:07.148623 11007 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 13785: Not authorized: Client connection negotiation failed: client connection to 127.10.191.195:40673: server requires authentication, but client does not have Kerberos credentials available
I20260620 23:11:07.148777 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 13785
I20260620 23:11:07.163609 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 13425
I20260620 23:11:07.164783 13487 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:11:07.264673 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 13425
2026-06-20T23:11:07Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3509 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997042893763-11007-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:11:07 dist-test-slave-jkbq krb5kdc[13930](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:11:07 dist-test-slave-jkbq krb5kdc[13930](info): set up 2 sockets
Jun 20 23:11:07 dist-test-slave-jkbq krb5kdc[13930](info): commencing operation
krb5kdc: starting...
W20260620 23:11:08.351171 11007 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Jun 20 23:11:08 dist-test-slave-jkbq krb5kdc[13930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:11:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:11:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.191.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
I20260620 23:11:08.540086 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:38015
--webserver_interface=127.10.191.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:41777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:38015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:11:08.852250 13943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:08.852519 13943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:08.852666 13943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:08.859742 13943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:11:08.859836 13943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:08.859922 13943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:08.859998 13943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:11:08.860066 13943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:11:08.870249 13943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:41777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:38015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:38015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:08.871537 13943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:08.872902 13943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:08.884371 13948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:08.884325 13949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:08.884441 13951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:08.885353 13943 server_base.cc:1034] running on GCE node
I20260620 23:11:08.886117 13943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:08.887630 13943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:08.888839 13943 hybrid_clock.cc:648] HybridClock initialized: now 1781997068888776 us; error 60 us; skew 500 ppm
I20260620 23:11:08.889147 13943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:11:08 dist-test-slave-jkbq krb5kdc[13930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997068, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:08.892212 13943 init.cc:377] Logged in from keytab as kudu/127.10.191.254@KRBTEST.COM (short username kudu)
I20260620 23:11:08.893698 13943 webserver.cc:460] Webserver started at http://127.10.191.254:39173/ using document root <none> and password file <none>
I20260620 23:11:08.894382 13943 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:08.894526 13943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:08.894847 13943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:11:08.897945 13943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "8b1ca6ea81b74c53b9d30e193f5f7476"
format_stamp: "Formatted at 2026-06-20 23:11:08 on dist-test-slave-jkbq"
server_key: "9ab5bef399f0a81532e3a8421cee34fd"
server_key_iv: "8fe68587c42009db498492b57b955cae"
server_key_version: "encryptionkey@0"
I20260620 23:11:08.898772 13943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "8b1ca6ea81b74c53b9d30e193f5f7476"
format_stamp: "Formatted at 2026-06-20 23:11:08 on dist-test-slave-jkbq"
server_key: "9ab5bef399f0a81532e3a8421cee34fd"
server_key_iv: "8fe68587c42009db498492b57b955cae"
server_key_version: "encryptionkey@0"
I20260620 23:11:08.904029 13943 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 23:11:08.907636 13958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:08.908964 13943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:11:08.909152 13943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "8b1ca6ea81b74c53b9d30e193f5f7476"
format_stamp: "Formatted at 2026-06-20 23:11:08 on dist-test-slave-jkbq"
server_key: "9ab5bef399f0a81532e3a8421cee34fd"
server_key_iv: "8fe68587c42009db498492b57b955cae"
server_key_version: "encryptionkey@0"
I20260620 23:11:08.909358 13943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:08.922768 13943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:08.926456 13943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:08.926724 13943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:08.944317 13943 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:38015
I20260620 23:11:08.944350 14009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:38015 every 8 connection(s)
I20260620 23:11:08.945778 13943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:11:08.946363 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 13943
I20260620 23:11:08.946584 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:11:08.947418 11007 external_mini_cluster.cc:1428] Setting key b09f94d9b3da823f18c9826836c41ed7
I20260620 23:11:08.950145 14010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 23:11:08 dist-test-slave-jkbq krb5kdc[13930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:11:08.960573 14010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b1ca6ea81b74c53b9d30e193f5f7476: Bootstrap starting.
I20260620 23:11:08.963946 14010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b1ca6ea81b74c53b9d30e193f5f7476: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:08.964946 14010 log.cc:824] T 00000000000000000000000000000000 P 8b1ca6ea81b74c53b9d30e193f5f7476: Log is configured to *not* fsync() on all Append() calls
I20260620 23:11:08.967895 14010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b1ca6ea81b74c53b9d30e193f5f7476: No bootstrap required, opened a new log
I20260620 23:11:08.968438 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:08.949180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44386 (local address 127.10.191.254:38015)
0620 23:11:08.949962 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:08.949973 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:08.950012 (+ 39us) server_negotiation.cc:407] Connection header received
0620 23:11:08.951704 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:08.951731 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:08.952095 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:08.952478 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:08.953979 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:08.955461 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:08.956664 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:08.957298 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:08.957508 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:08.961314 (+ 3806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:08.961321 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:08.961335 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:08.961368 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:08.964553 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:08.965204 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:08.965216 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:08.965222 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:08.965428 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:08.965836 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:08.965845 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:08.965850 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:08.966424 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:08.966613 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:08.967100 (+ 487us) server_negotiation.cc:299] Negotiation successful
0620 23:11:08.967411 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":376,"threads_started":1}
I20260620 23:11:08.972262 11007 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:11:08.978384 14010 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b1ca6ea81b74c53b9d30e193f5f7476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b1ca6ea81b74c53b9d30e193f5f7476" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 38015 } }
I20260620 23:11:08.978811 14010 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b1ca6ea81b74c53b9d30e193f5f7476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:08.978957 14010 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b1ca6ea81b74c53b9d30e193f5f7476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b1ca6ea81b74c53b9d30e193f5f7476, State: Initialized, Role: FOLLOWER
I20260620 23:11:08.979628 14010 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b1ca6ea81b74c53b9d30e193f5f7476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b1ca6ea81b74c53b9d30e193f5f7476" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 38015 } }
I20260620 23:11:08.979887 14010 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b1ca6ea81b74c53b9d30e193f5f7476 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:11:08.980014 14010 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b1ca6ea81b74c53b9d30e193f5f7476 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:11:08.980172 14010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b1ca6ea81b74c53b9d30e193f5f7476 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:08.982234 14010 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b1ca6ea81b74c53b9d30e193f5f7476 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b1ca6ea81b74c53b9d30e193f5f7476" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 38015 } }
I20260620 23:11:08.982726 14010 leader_election.cc:302] T 00000000000000000000000000000000 P 8b1ca6ea81b74c53b9d30e193f5f7476 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b1ca6ea81b74c53b9d30e193f5f7476; no voters:
I20260620 23:11:08.983345 14010 leader_election.cc:288] T 00000000000000000000000000000000 P 8b1ca6ea81b74c53b9d30e193f5f7476 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:11:08.983479 14016 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b1ca6ea81b74c53b9d30e193f5f7476 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:11:08.984724 14016 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b1ca6ea81b74c53b9d30e193f5f7476 [term 1 LEADER]: Becoming Leader. State: Replica: 8b1ca6ea81b74c53b9d30e193f5f7476, State: Running, Role: LEADER
I20260620 23:11:08.985316 14016 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b1ca6ea81b74c53b9d30e193f5f7476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b1ca6ea81b74c53b9d30e193f5f7476" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 38015 } }
I20260620 23:11:08.985702 14010 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b1ca6ea81b74c53b9d30e193f5f7476 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:11:08.988641 14018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b1ca6ea81b74c53b9d30e193f5f7476 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b1ca6ea81b74c53b9d30e193f5f7476. Latest consensus state: current_term: 1 leader_uuid: "8b1ca6ea81b74c53b9d30e193f5f7476" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b1ca6ea81b74c53b9d30e193f5f7476" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 38015 } } }
I20260620 23:11:08.989060 14018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b1ca6ea81b74c53b9d30e193f5f7476 [sys.catalog]: This master's current role is: LEADER
I20260620 23:11:08.989713 14017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b1ca6ea81b74c53b9d30e193f5f7476 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b1ca6ea81b74c53b9d30e193f5f7476" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b1ca6ea81b74c53b9d30e193f5f7476" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 38015 } } }
I20260620 23:11:08.990038 14017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b1ca6ea81b74c53b9d30e193f5f7476 [sys.catalog]: This master's current role is: LEADER
I20260620 23:11:09.000075 14022 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:11:09.005955 14022 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:11:09.017541 14022 catalog_manager.cc:1347] Generated new cluster ID: 302984ad60934961b0f00c8bc4ade277
I20260620 23:11:09.017800 14022 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:11:09.038514 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:09.029671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44402 (local address 127.10.191.254:38015)
0620 23:11:09.029842 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:09.029849 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:09.030382 (+ 533us) server_negotiation.cc:407] Connection header received
0620 23:11:09.030546 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:09.030552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:09.030639 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:09.030750 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:09.031877 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:09.033028 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:09.034729 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:09.035477 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:09.035624 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:09.036756 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:09.036767 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:09.036771 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:11:09.037680 (+ 909us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:09.037747 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:11:09.037781 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:09.037894 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:09.038039 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 23:11:09.038127 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 23:11:09.040248 14022 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:11:09.042049 14022 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:11:09.062520 14022 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b1ca6ea81b74c53b9d30e193f5f7476: Generated new TSK 0
I20260620 23:11:09.063742 14022 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:11:09.078943 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 13943
I20260620 23:11:09.080142 14005 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:11:09.169947 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 13943
2026-06-20T23:11:09Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1898 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:11:09 dist-test-slave-jkbq krb5kdc[14045](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:11:09 dist-test-slave-jkbq krb5kdc[14045](info): set up 2 sockets
Jun 20 23:11:09 dist-test-slave-jkbq krb5kdc[14045](info): commencing operation
krb5kdc: starting...
W20260620 23:11:10.283627 11007 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Jun 20 23:11:10 dist-test-slave-jkbq krb5kdc[14045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:11:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:11:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.191.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
I20260620 23:11:10.473878 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:42833
--webserver_interface=127.10.191.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:35069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:42833
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:11:10.785835 14058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:10.786109 14058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:10.786269 14058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:10.793440 14058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:11:10.793537 14058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:10.793619 14058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:10.793715 14058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:11:10.793797 14058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:11:10.803515 14058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:35069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:42833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:42833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:10.804868 14058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:10.806193 14058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:10.817662 14063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:10.817689 14066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:10.817639 14064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:10.818022 14058 server_base.cc:1034] running on GCE node
I20260620 23:11:10.819099 14058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:10.820349 14058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:10.821589 14058 hybrid_clock.cc:648] HybridClock initialized: now 1781997070821541 us; error 38 us; skew 500 ppm
I20260620 23:11:10.822022 14058 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:11:10 dist-test-slave-jkbq krb5kdc[14045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997070, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:10.824965 14058 init.cc:377] Logged in from keytab as kudu/127.10.191.254@KRBTEST.COM (short username kudu)
I20260620 23:11:10.826282 14058 webserver.cc:460] Webserver started at http://127.10.191.254:36415/ using document root <none> and password file <none>
I20260620 23:11:10.826984 14058 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:10.827127 14058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:10.827421 14058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:11:10.830211 14058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "ab32129a839e497cbd9a002aa617ca39"
format_stamp: "Formatted at 2026-06-20 23:11:10 on dist-test-slave-jkbq"
server_key: "0d4eccbf78feef0942408b3f1a8534b2"
server_key_iv: "63138e8dab0de5e8ef7a6b5d57ae9b61"
server_key_version: "encryptionkey@0"
I20260620 23:11:10.831030 14058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "ab32129a839e497cbd9a002aa617ca39"
format_stamp: "Formatted at 2026-06-20 23:11:10 on dist-test-slave-jkbq"
server_key: "0d4eccbf78feef0942408b3f1a8534b2"
server_key_iv: "63138e8dab0de5e8ef7a6b5d57ae9b61"
server_key_version: "encryptionkey@0"
I20260620 23:11:10.836205 14058 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 23:11:10.839516 14073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:10.840797 14058 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:11:10.840981 14058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "ab32129a839e497cbd9a002aa617ca39"
format_stamp: "Formatted at 2026-06-20 23:11:10 on dist-test-slave-jkbq"
server_key: "0d4eccbf78feef0942408b3f1a8534b2"
server_key_iv: "63138e8dab0de5e8ef7a6b5d57ae9b61"
server_key_version: "encryptionkey@0"
I20260620 23:11:10.841167 14058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:10.876128 14058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:10.879979 14058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:10.880262 14058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:10.898211 14058 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:42833
I20260620 23:11:10.898243 14124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:42833 every 8 connection(s)
I20260620 23:11:10.899648 14058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:11:10.901440 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 14058
I20260620 23:11:10.901654 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:11:10.902282 11007 external_mini_cluster.cc:1428] Setting key 2764e69552d4c523686aa11530af1e98
I20260620 23:11:10.903545 14125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 23:11:10 dist-test-slave-jkbq krb5kdc[14045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:11:10.917184 14125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab32129a839e497cbd9a002aa617ca39: Bootstrap starting.
I20260620 23:11:10.921329 14125 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab32129a839e497cbd9a002aa617ca39: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:10.922595 14125 log.cc:824] T 00000000000000000000000000000000 P ab32129a839e497cbd9a002aa617ca39: Log is configured to *not* fsync() on all Append() calls
I20260620 23:11:10.925525 14128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:10.904322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53722 (local address 127.10.191.254:42833)
0620 23:11:10.905065 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:10.905080 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:10.905128 (+ 48us) server_negotiation.cc:407] Connection header received
0620 23:11:10.907044 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:10.907073 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:10.907591 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:10.908022 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:10.909238 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:10.911275 (+ 2037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:10.912466 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:10.913461 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:10.913658 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:10.917721 (+ 4063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:10.917732 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:10.917751 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:10.917791 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:10.921593 (+ 3802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:10.922246 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:10.922256 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:10.922267 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:10.922478 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:10.922919 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:10.922923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:10.922925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:10.923447 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:10.923775 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:10.924179 (+ 404us) server_negotiation.cc:299] Negotiation successful
0620 23:11:10.924583 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":304,"threads_started":1}
I20260620 23:11:10.925575 14125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab32129a839e497cbd9a002aa617ca39: No bootstrap required, opened a new log
I20260620 23:11:10.933449 14125 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab32129a839e497cbd9a002aa617ca39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab32129a839e497cbd9a002aa617ca39" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 42833 } }
I20260620 23:11:10.933832 14125 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab32129a839e497cbd9a002aa617ca39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:10.933943 14125 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab32129a839e497cbd9a002aa617ca39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab32129a839e497cbd9a002aa617ca39, State: Initialized, Role: FOLLOWER
I20260620 23:11:10.934480 14125 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab32129a839e497cbd9a002aa617ca39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab32129a839e497cbd9a002aa617ca39" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 42833 } }
I20260620 23:11:10.934713 14125 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab32129a839e497cbd9a002aa617ca39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:11:10.934859 14125 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab32129a839e497cbd9a002aa617ca39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:11:10.935089 14125 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab32129a839e497cbd9a002aa617ca39 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:10.937430 14125 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab32129a839e497cbd9a002aa617ca39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab32129a839e497cbd9a002aa617ca39" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 42833 } }
I20260620 23:11:10.937917 14125 leader_election.cc:302] T 00000000000000000000000000000000 P ab32129a839e497cbd9a002aa617ca39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab32129a839e497cbd9a002aa617ca39; no voters:
I20260620 23:11:10.938437 14125 leader_election.cc:288] T 00000000000000000000000000000000 P ab32129a839e497cbd9a002aa617ca39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:11:10.938576 14130 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab32129a839e497cbd9a002aa617ca39 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:11:10.939275 14130 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab32129a839e497cbd9a002aa617ca39 [term 1 LEADER]: Becoming Leader. State: Replica: ab32129a839e497cbd9a002aa617ca39, State: Running, Role: LEADER
I20260620 23:11:10.939793 14130 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab32129a839e497cbd9a002aa617ca39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab32129a839e497cbd9a002aa617ca39" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 42833 } }
I20260620 23:11:10.940512 14125 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab32129a839e497cbd9a002aa617ca39 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:11:10.942384 14131 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab32129a839e497cbd9a002aa617ca39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab32129a839e497cbd9a002aa617ca39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab32129a839e497cbd9a002aa617ca39" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 42833 } } }
I20260620 23:11:10.942842 14131 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab32129a839e497cbd9a002aa617ca39 [sys.catalog]: This master's current role is: LEADER
I20260620 23:11:10.947099 14132 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab32129a839e497cbd9a002aa617ca39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab32129a839e497cbd9a002aa617ca39. Latest consensus state: current_term: 1 leader_uuid: "ab32129a839e497cbd9a002aa617ca39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab32129a839e497cbd9a002aa617ca39" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 42833 } } }
I20260620 23:11:10.947592 14132 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab32129a839e497cbd9a002aa617ca39 [sys.catalog]: This master's current role is: LEADER
I20260620 23:11:10.954909 14139 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:11:10.959242 14139 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:11:10.967264 14139 catalog_manager.cc:1347] Generated new cluster ID: df45f8ef155b46259698b208945c083b
I20260620 23:11:10.967365 14139 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:11:10.978886 14139 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:11:10.980059 14139 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:11:10.989285 14139 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab32129a839e497cbd9a002aa617ca39: Generated new TSK 0
I20260620 23:11:10.990015 14139 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.191.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab.
Entry for principal kudu/127.10.191.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab.
I20260620 23:11:11.036808 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.193:0
--local_ip_for_outbound_sockets=127.10.191.193
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:42833
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:35069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:11:11.358615 14151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:11.358981 14151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:11.359161 14151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:11.366518 14151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:11.366626 14151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:11.366849 14151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.193
I20260620 23:11:11.376783 14151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:35069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:42833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.193
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:11.378134 14151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:11.379568 14151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:11.393782 14156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:11.393821 14157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:11.393847 14159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:11.395100 14151 server_base.cc:1034] running on GCE node
I20260620 23:11:11.395742 14151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:11.396730 14151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:11.398013 14151 hybrid_clock.cc:648] HybridClock initialized: now 1781997071397955 us; error 61 us; skew 500 ppm
I20260620 23:11:11.398360 14151 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:11:11 dist-test-slave-jkbq krb5kdc[14045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997071, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:11.401856 14151 init.cc:377] Logged in from keytab as kudu/127.10.191.193@KRBTEST.COM (short username kudu)
I20260620 23:11:11.403409 14151 webserver.cc:460] Webserver started at http://127.10.191.193:39205/ using document root <none> and password file <none>
I20260620 23:11:11.404129 14151 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:11.404286 14151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:11.404596 14151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:11:11.407644 14151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-0/data/instance:
uuid: "f9e26daaa06c4d47b93d1dbe0cf7ca6e"
format_stamp: "Formatted at 2026-06-20 23:11:11 on dist-test-slave-jkbq"
server_key: "6d1fe4bf59ff47bb6905129b35792d32"
server_key_iv: "0466b80fef4db1565147c522f73158a8"
server_key_version: "encryptionkey@0"
I20260620 23:11:11.408478 14151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance:
uuid: "f9e26daaa06c4d47b93d1dbe0cf7ca6e"
format_stamp: "Formatted at 2026-06-20 23:11:11 on dist-test-slave-jkbq"
server_key: "6d1fe4bf59ff47bb6905129b35792d32"
server_key_iv: "0466b80fef4db1565147c522f73158a8"
server_key_version: "encryptionkey@0"
I20260620 23:11:11.414096 14151 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 23:11:11.417798 14166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:11.419232 14151 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 23:11:11.419420 14151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-0/wal
uuid: "f9e26daaa06c4d47b93d1dbe0cf7ca6e"
format_stamp: "Formatted at 2026-06-20 23:11:11 on dist-test-slave-jkbq"
server_key: "6d1fe4bf59ff47bb6905129b35792d32"
server_key_iv: "0466b80fef4db1565147c522f73158a8"
server_key_version: "encryptionkey@0"
I20260620 23:11:11.419624 14151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:11.431311 14151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:11.435434 14151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:11.435737 14151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:11.436697 14151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:11:11.438037 14151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:11:11.438143 14151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:11.438288 14151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:11:11.438362 14151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:11.471577 14151 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.193:36813
I20260620 23:11:11.471621 14278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.193:36813 every 8 connection(s)
I20260620 23:11:11.472898 14151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
I20260620 23:11:11.474574 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 14151
I20260620 23:11:11.474781 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance
I20260620 23:11:11.475622 11007 external_mini_cluster.cc:1428] Setting key 4735ce9573d56d91432f38b11f530718
Jun 20 23:11:11 dist-test-slave-jkbq krb5kdc[14045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997071, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.191.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.194@KRBTEST.COM" created.
I20260620 23:11:11.498495 14281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:11.475672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:44955 (local address 127.10.191.254:42833)
0620 23:11:11.476188 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:11.476193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:11.477416 (+ 1223us) server_negotiation.cc:407] Connection header received
0620 23:11:11.479295 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:11.479304 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:11.479393 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:11.479518 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:11.480984 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:11.482186 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:11.484349 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:11.484738 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:11.484835 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:11.492543 (+ 7708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:11.492549 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:11.492552 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:11.492585 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:11.495399 (+ 2814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:11.496168 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:11.496180 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:11.496189 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:11.496306 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:11.496753 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:11.496760 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:11.496767 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:11.497029 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:11.497185 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:11.498034 (+ 849us) server_negotiation.cc:299] Negotiation successful
0620 23:11:11.498201 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":279,"threads_started":1}
I20260620 23:11:11.500180 14282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:11.476380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:42833 (local address 127.10.191.193:44955)
0620 23:11:11.477192 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:11.477245 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:11.479018 (+ 1773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:11.479714 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:11.479722 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:11.480214 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:11.480795 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:11.480811 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:11.482399 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:11.482407 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:11.484211 (+ 1804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:11.484228 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:11.484969 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:11.484979 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:11.485311 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:11.489289 (+ 3978us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:11.489327 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:11.492283 (+ 2956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:11.495563 (+ 3280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:11.495572 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:11.495590 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:11.496026 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:11.496422 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:11.496428 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:11.496432 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:11.496628 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:11.497171 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:11.497179 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:11.497600 (+ 421us) client_negotiation.cc:769] Sending connection context
0620 23:11:11.497925 (+ 325us) client_negotiation.cc:240] Negotiation successful
0620 23:11:11.498563 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":343,"threads_started":1}
I20260620 23:11:11.501832 14279 heartbeater.cc:347] Connected to a master server at 127.10.191.254:42833
I20260620 23:11:11.502194 14279 heartbeater.cc:464] Registering TS with master...
I20260620 23:11:11.503032 14279 heartbeater.cc:511] Master 127.10.191.254:42833 requested a full tablet report, sending...
I20260620 23:11:11.505281 14090 ts_manager.cc:194] Registered new tserver with Master: f9e26daaa06c4d47b93d1dbe0cf7ca6e (127.10.191.193:36813)
I20260620 23:11:11.506764 14090 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.193@KRBTEST.COM'} at 127.10.191.193:44955
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab.
Entry for principal kudu/127.10.191.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab.
I20260620 23:11:11.521713 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.194:0
--local_ip_for_outbound_sockets=127.10.191.194
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:42833
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:35069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:11:11.843282 14285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:11.843611 14285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:11.843778 14285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:11.851344 14285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:11.851500 14285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:11.851682 14285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.194
I20260620 23:11:11.861255 14285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:35069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:42833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.194
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:11.862576 14285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:11.863909 14285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:11.876876 14290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:11.876835 14293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:11.876838 14291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:11.877730 14285 server_base.cc:1034] running on GCE node
I20260620 23:11:11.878398 14285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:11.879321 14285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:11.880539 14285 hybrid_clock.cc:648] HybridClock initialized: now 1781997071880499 us; error 38 us; skew 500 ppm
I20260620 23:11:11.880841 14285 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:11:11 dist-test-slave-jkbq krb5kdc[14045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997071, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:11.883934 14285 init.cc:377] Logged in from keytab as kudu/127.10.191.194@KRBTEST.COM (short username kudu)
I20260620 23:11:11.885370 14285 webserver.cc:460] Webserver started at http://127.10.191.194:36581/ using document root <none> and password file <none>
I20260620 23:11:11.886021 14285 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:11.886168 14285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:11.886469 14285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:11:11.889344 14285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-1/data/instance:
uuid: "2227ca075ba44c6a9904373d869cf048"
format_stamp: "Formatted at 2026-06-20 23:11:11 on dist-test-slave-jkbq"
server_key: "29d255d68dcecbede2b15dd44c1ee61e"
server_key_iv: "7ac4bf39ed7eaea683cf3361206367c7"
server_key_version: "encryptionkey@0"
I20260620 23:11:11.890102 14285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance:
uuid: "2227ca075ba44c6a9904373d869cf048"
format_stamp: "Formatted at 2026-06-20 23:11:11 on dist-test-slave-jkbq"
server_key: "29d255d68dcecbede2b15dd44c1ee61e"
server_key_iv: "7ac4bf39ed7eaea683cf3361206367c7"
server_key_version: "encryptionkey@0"
I20260620 23:11:11.894994 14285 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 23:11:11.898296 14300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:11.899641 14285 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 23:11:11.899811 14285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-1/wal
uuid: "2227ca075ba44c6a9904373d869cf048"
format_stamp: "Formatted at 2026-06-20 23:11:11 on dist-test-slave-jkbq"
server_key: "29d255d68dcecbede2b15dd44c1ee61e"
server_key_iv: "7ac4bf39ed7eaea683cf3361206367c7"
server_key_version: "encryptionkey@0"
I20260620 23:11:11.900007 14285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:11.922691 14285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:11.926680 14285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:11.927038 14285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:11.928001 14285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:11:11.929301 14285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:11:11.929409 14285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:11.929553 14285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:11:11.929625 14285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:11.962913 14285 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.194:37503
I20260620 23:11:11.963004 14412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.194:37503 every 8 connection(s)
I20260620 23:11:11.964196 14285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
I20260620 23:11:11.969986 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 14285
I20260620 23:11:11.970167 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance
I20260620 23:11:11.970696 11007 external_mini_cluster.cc:1428] Setting key 03f87ffca7e4e1c7c89b77fe6634cc34
Jun 20 23:11:11 dist-test-slave-jkbq krb5kdc[14045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997071, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:11:11.986068 14281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:11.966792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:35999 (local address 127.10.191.254:42833)
0620 23:11:11.967035 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:11.967040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:11.968212 (+ 1172us) server_negotiation.cc:407] Connection header received
0620 23:11:11.970012 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:11.970020 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:11.970111 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:11.970230 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:11.971998 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:11.972957 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:11.975475 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:11.975861 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:11.975962 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:11.980377 (+ 4415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:11.980382 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:11.980385 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:11.980414 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:11.982826 (+ 2412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:11.983780 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:11.983783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:11.983785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:11.983911 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:11.984380 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:11.984383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:11.984385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:11.984624 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:11.984727 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:11.985583 (+ 856us) server_negotiation.cc:299] Negotiation successful
0620 23:11:11.985789 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260620 23:11:11.987636 14415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:11.967309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:42833 (local address 127.10.191.194:35999)
0620 23:11:11.968047 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:11.968085 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:11.969705 (+ 1620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:11.970455 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:11.970463 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:11.971119 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:11.971823 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:11.971845 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:11.973100 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:11.973105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:11.975339 (+ 2234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:11.975357 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:11.976091 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:11.976100 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:11.976430 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:11.977187 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:11.977229 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:11.980148 (+ 2919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:11.983025 (+ 2877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:11.983036 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:11.983056 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:11.983650 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:11.984014 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:11.984021 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:11.984028 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:11.984274 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:11.984755 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:11.984766 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:11.985212 (+ 446us) client_negotiation.cc:769] Sending connection context
0620 23:11:11.985506 (+ 294us) client_negotiation.cc:240] Negotiation successful
0620 23:11:11.986089 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":300,"threads_started":1}
I20260620 23:11:11.989302 14413 heartbeater.cc:347] Connected to a master server at 127.10.191.254:42833
I20260620 23:11:11.989620 14413 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.10.191.195@KRBTEST.COM; defaulting to no policy
I20260620 23:11:11.990374 14413 heartbeater.cc:511] Master 127.10.191.254:42833 requested a full tablet report, sending...
I20260620 23:11:11.991888 14090 ts_manager.cc:194] Registered new tserver with Master: 2227ca075ba44c6a9904373d869cf048 (127.10.191.194:37503)
I20260620 23:11:11.992693 14090 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.194@KRBTEST.COM'} at 127.10.191.194:35999
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab.
Entry for principal kudu/127.10.191.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab.
I20260620 23:11:12.014392 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.195:0
--local_ip_for_outbound_sockets=127.10.191.195
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:42833
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:35069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:11:12.331408 14418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:12.331667 14418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:12.331813 14418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:12.338737 14418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:12.338835 14418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:12.339020 14418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.195
I20260620 23:11:12.348537 14418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:35069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:42833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.195
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:12.349802 14418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:12.351114 14418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:12.363423 14423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:12.363778 14426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:12.363969 14424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:12.364774 14418 server_base.cc:1034] running on GCE node
I20260620 23:11:12.365350 14418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:12.366243 14418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:12.367463 14418 hybrid_clock.cc:648] HybridClock initialized: now 1781997072367413 us; error 64 us; skew 500 ppm
I20260620 23:11:12.367775 14418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:11:12 dist-test-slave-jkbq krb5kdc[14045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997072, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:12.370805 14418 init.cc:377] Logged in from keytab as kudu/127.10.191.195@KRBTEST.COM (short username kudu)
I20260620 23:11:12.372123 14418 webserver.cc:460] Webserver started at http://127.10.191.195:41033/ using document root <none> and password file <none>
I20260620 23:11:12.372779 14418 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:12.372922 14418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:12.373208 14418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:11:12.376073 14418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-2/data/instance:
uuid: "81ad5b84469d452ca0c117168dbd81e5"
format_stamp: "Formatted at 2026-06-20 23:11:12 on dist-test-slave-jkbq"
server_key: "2a827ce0bd6f90e4f3cca1c506c6344b"
server_key_iv: "4cc06e259165fe9d549372047127d590"
server_key_version: "encryptionkey@0"
I20260620 23:11:12.376830 14418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance:
uuid: "81ad5b84469d452ca0c117168dbd81e5"
format_stamp: "Formatted at 2026-06-20 23:11:12 on dist-test-slave-jkbq"
server_key: "2a827ce0bd6f90e4f3cca1c506c6344b"
server_key_iv: "4cc06e259165fe9d549372047127d590"
server_key_version: "encryptionkey@0"
I20260620 23:11:12.381743 14418 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260620 23:11:12.385910 14433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:12.387449 14418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:11:12.387578 14418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-2/wal
uuid: "81ad5b84469d452ca0c117168dbd81e5"
format_stamp: "Formatted at 2026-06-20 23:11:12 on dist-test-slave-jkbq"
server_key: "2a827ce0bd6f90e4f3cca1c506c6344b"
server_key_iv: "4cc06e259165fe9d549372047127d590"
server_key_version: "encryptionkey@0"
I20260620 23:11:12.387734 14418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:12.404106 14418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:12.408321 14418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:12.408564 14418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:12.409453 14418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:11:12.410734 14418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:11:12.410813 14418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:12.410969 14418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:11:12.411031 14418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:12.445175 14418 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.195:38333
I20260620 23:11:12.445202 14545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.195:38333 every 8 connection(s)
I20260620 23:11:12.446491 14418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
I20260620 23:11:12.451462 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 14418
I20260620 23:11:12.451660 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance
I20260620 23:11:12.452162 11007 external_mini_cluster.cc:1428] Setting key 00a856ca9745baced9e68bef2cec1e61
Jun 20 23:11:12 dist-test-slave-jkbq krb5kdc[14045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997072, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:11:12.466101 14281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:12.449047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:45105 (local address 127.10.191.254:42833)
0620 23:11:12.449278 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:12.449285 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:12.450494 (+ 1209us) server_negotiation.cc:407] Connection header received
0620 23:11:12.452494 (+ 2000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:12.452501 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:12.452614 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:12.452761 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:12.454391 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:12.455328 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:12.457005 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:12.457406 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:12.457534 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:12.460824 (+ 3290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:12.460830 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:12.460833 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:12.460865 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:12.463326 (+ 2461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:12.463996 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:12.464003 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:12.464007 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:12.464121 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:12.464518 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:12.464525 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:12.464528 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:12.464780 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:12.464932 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:12.465539 (+ 607us) server_negotiation.cc:299] Negotiation successful
0620 23:11:12.465743 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 23:11:12.467216 14548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:12.449620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:42833 (local address 127.10.191.195:45105)
0620 23:11:12.450338 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:12.450377 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:12.452183 (+ 1806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:12.453035 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:12.453044 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:12.453570 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:12.454222 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:12.454241 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:12.455485 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:12.455489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:12.456868 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:12.456880 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:12.457616 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:12.457624 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:12.457868 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:12.458475 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:12.458496 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:12.460627 (+ 2131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:12.463463 (+ 2836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:12.463471 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:12.463487 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:12.463880 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:12.464227 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:12.464232 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:12.464236 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:12.464414 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:12.464909 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:12.464917 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:12.465218 (+ 301us) client_negotiation.cc:769] Sending connection context
0620 23:11:12.465466 (+ 248us) client_negotiation.cc:240] Negotiation successful
0620 23:11:12.465968 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":270,"threads_started":1}
I20260620 23:11:12.468492 14546 heartbeater.cc:347] Connected to a master server at 127.10.191.254:42833
I20260620 23:11:12.468734 14546 heartbeater.cc:464] Registering TS with master...
I20260620 23:11:12.469395 14546 heartbeater.cc:511] Master 127.10.191.254:42833 requested a full tablet report, sending...
I20260620 23:11:12.470621 14090 ts_manager.cc:194] Registered new tserver with Master: 81ad5b84469d452ca0c117168dbd81e5 (127.10.191.195:38333)
I20260620 23:11:12.471544 14090 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.195@KRBTEST.COM'} at 127.10.191.195:45105
I20260620 23:11:12.479576 11007 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 23:11:12 dist-test-slave-jkbq krb5kdc[14045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997072, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
I20260620 23:11:12.509948 14279 heartbeater.cc:503] Master 127.10.191.254:42833 was elected leader, sending a full tablet report...
Jun 20 23:11:12 dist-test-slave-jkbq krb5kdc[14045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997072, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:11:12.525775 14281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:12.512354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53728 (local address 127.10.191.254:42833)
0620 23:11:12.512538 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:12.512546 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:12.512912 (+ 366us) server_negotiation.cc:407] Connection header received
0620 23:11:12.513116 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:12.513123 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:12.513183 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:12.513284 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:12.514660 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:12.515432 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:12.516694 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:12.517095 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:12.517227 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:12.520940 (+ 3713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:12.520945 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:12.520948 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:12.520977 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:12.523317 (+ 2340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:12.523928 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:12.523935 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:12.523938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:12.524059 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:12.524440 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:12.524445 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:12.524448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:12.524744 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:12.524989 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:12.525273 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 23:11:12.525471 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 23:11:12.526254 14090 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53728
I20260620 23:11:12.528435 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 14151
Jun 20 23:11:12 dist-test-slave-jkbq krb5kdc[14045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997072, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.10.191.193@KRBTEST.COM
I20260620 23:11:12.543684 14557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:12.529529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38538 (local address 127.10.191.193:36813)
0620 23:11:12.530250 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:12.530257 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:12.530281 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:11:12.530333 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:12.530340 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:12.530538 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:12.530668 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:12.531872 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:12.532877 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:12.534061 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:12.534600 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:12.534713 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:12.538420 (+ 3707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:12.538430 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:12.538443 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:12.538477 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:12.541244 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:12.541856 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:12.541863 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:12.541865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:12.541981 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:12.542313 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:12.542316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:12.542318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:12.542619 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:12.542840 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:12.543232 (+ 392us) server_negotiation.cc:299] Negotiation successful
0620 23:11:12.543413 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":432,"threads_started":1}
W20260620 23:11:12.544976 14274 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:38538
W20260620 23:11:12.546146 11007 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 14151: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:11:12.546332 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 14151
I20260620 23:11:12.561805 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 14285
Jun 20 23:11:12 dist-test-slave-jkbq krb5kdc[14045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997072, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.10.191.194@KRBTEST.COM
I20260620 23:11:12.577436 14558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:12.562831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59380 (local address 127.10.191.194:37503)
0620 23:11:12.563457 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:12.563464 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:12.563491 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:11:12.563550 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:12.563557 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:12.563784 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:12.563913 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:12.565037 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:12.566033 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:12.567280 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:12.567814 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:12.567923 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:12.571849 (+ 3926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:12.571860 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:12.571874 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:12.571921 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:12.574678 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:12.575372 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:12.575379 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:12.575381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:12.575500 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:12.575876 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:12.575885 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:12.575887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:12.576209 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:12.576507 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:12.576813 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 23:11:12.577083 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":302,"threads_started":1}
W20260620 23:11:12.578619 14408 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:59380
W20260620 23:11:12.579716 11007 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 14285: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:11:12.579856 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 14285
I20260620 23:11:12.595022 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 14418
Jun 20 23:11:12 dist-test-slave-jkbq krb5kdc[14045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997072, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.10.191.195@KRBTEST.COM
I20260620 23:11:12.611392 14559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:12.596007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43832 (local address 127.10.191.195:38333)
0620 23:11:12.596585 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:12.596592 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:12.596620 (+ 28us) server_negotiation.cc:407] Connection header received
0620 23:11:12.596678 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:12.596684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:12.596900 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:12.597075 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:12.598409 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:12.599545 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:12.600774 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:12.601328 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:12.601464 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:12.605887 (+ 4423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:12.605898 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:12.605914 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:12.605947 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:12.608842 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:12.609435 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:12.609442 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:12.609444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:12.609560 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:12.609932 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:12.609936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:12.609938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:12.610288 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:12.610514 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:12.610799 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 23:11:12.611095 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":296,"threads_started":1}
W20260620 23:11:12.612633 14541 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43832
W20260620 23:11:12.613749 11007 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 14418: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:11:12.613893 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 14418
I20260620 23:11:12.629145 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 14058
I20260620 23:11:12.630286 14120 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:11:12.732686 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 14058
2026-06-20T23:11:12Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3567 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:11:12 dist-test-slave-jkbq krb5kdc[14563](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:11:12 dist-test-slave-jkbq krb5kdc[14563](info): set up 2 sockets
Jun 20 23:11:12 dist-test-slave-jkbq krb5kdc[14563](info): commencing operation
krb5kdc: starting...
W20260620 23:11:13.835214 11007 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Jun 20 23:11:13 dist-test-slave-jkbq krb5kdc[14563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:11:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:11:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.191.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
I20260620 23:11:14.024209 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:37823
--webserver_interface=127.10.191.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:43855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:37823
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:11:14.341446 14576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:14.341732 14576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:14.341823 14576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:14.348779 14576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:11:14.348879 14576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:14.348925 14576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:14.348973 14576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:11:14.349020 14576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:11:14.358390 14576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:43855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:37823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:37823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:14.359632 14576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:14.360890 14576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:14.372385 14584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:14.372448 14582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:14.372457 14581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:14.373564 14576 server_base.cc:1034] running on GCE node
I20260620 23:11:14.374393 14576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:14.375857 14576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:14.377058 14576 hybrid_clock.cc:648] HybridClock initialized: now 1781997074377017 us; error 38 us; skew 500 ppm
I20260620 23:11:14.377411 14576 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:11:14 dist-test-slave-jkbq krb5kdc[14563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997074, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:14.380509 14576 init.cc:377] Logged in from keytab as kudu/127.10.191.254@KRBTEST.COM (short username kudu)
I20260620 23:11:14.382050 14576 webserver.cc:460] Webserver started at http://127.10.191.254:37683/ using document root <none> and password file <none>
I20260620 23:11:14.382752 14576 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:14.382920 14576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:14.383235 14576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:11:14.386080 14576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "a268b276f30143818e61a95f0a39d40d"
format_stamp: "Formatted at 2026-06-20 23:11:14 on dist-test-slave-jkbq"
server_key: "ed92ff2e3228d11a61f70e5271251c60"
server_key_iv: "c53483ad55b835fcc3253f1048118b8b"
server_key_version: "encryptionkey@0"
I20260620 23:11:14.386952 14576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "a268b276f30143818e61a95f0a39d40d"
format_stamp: "Formatted at 2026-06-20 23:11:14 on dist-test-slave-jkbq"
server_key: "ed92ff2e3228d11a61f70e5271251c60"
server_key_iv: "c53483ad55b835fcc3253f1048118b8b"
server_key_version: "encryptionkey@0"
I20260620 23:11:14.392345 14576 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 23:11:14.396387 14591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:14.397708 14576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:11:14.397889 14576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "a268b276f30143818e61a95f0a39d40d"
format_stamp: "Formatted at 2026-06-20 23:11:14 on dist-test-slave-jkbq"
server_key: "ed92ff2e3228d11a61f70e5271251c60"
server_key_iv: "c53483ad55b835fcc3253f1048118b8b"
server_key_version: "encryptionkey@0"
I20260620 23:11:14.398080 14576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:14.419016 14576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:14.423133 14576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:14.423408 14576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:14.441425 14576 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:37823
I20260620 23:11:14.441457 14642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:37823 every 8 connection(s)
I20260620 23:11:14.442777 14576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:11:14.446460 14643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:11:14.452454 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 14576
I20260620 23:11:14.452682 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:11:14.453470 11007 external_mini_cluster.cc:1428] Setting key c7b8d5041802fb304bdd24785b0f364a
I20260620 23:11:14.457401 14643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a268b276f30143818e61a95f0a39d40d: Bootstrap starting.
I20260620 23:11:14.461093 14643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a268b276f30143818e61a95f0a39d40d: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:14.462220 14643 log.cc:824] T 00000000000000000000000000000000 P a268b276f30143818e61a95f0a39d40d: Log is configured to *not* fsync() on all Append() calls
I20260620 23:11:14.465013 14643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a268b276f30143818e61a95f0a39d40d: No bootstrap required, opened a new log
Jun 20 23:11:14 dist-test-slave-jkbq krb5kdc[14563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:11:14.474037 14643 raft_consensus.cc:348] T 00000000000000000000000000000000 P a268b276f30143818e61a95f0a39d40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a268b276f30143818e61a95f0a39d40d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 37823 } }
I20260620 23:11:14.474493 14643 raft_consensus.cc:374] T 00000000000000000000000000000000 P a268b276f30143818e61a95f0a39d40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:14.474607 14643 raft_consensus.cc:729] T 00000000000000000000000000000000 P a268b276f30143818e61a95f0a39d40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a268b276f30143818e61a95f0a39d40d, State: Initialized, Role: FOLLOWER
I20260620 23:11:14.475229 14643 consensus_queue.cc:260] T 00000000000000000000000000000000 P a268b276f30143818e61a95f0a39d40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a268b276f30143818e61a95f0a39d40d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 37823 } }
I20260620 23:11:14.475486 14643 raft_consensus.cc:388] T 00000000000000000000000000000000 P a268b276f30143818e61a95f0a39d40d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:11:14.475623 14643 raft_consensus.cc:482] T 00000000000000000000000000000000 P a268b276f30143818e61a95f0a39d40d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:11:14.475821 14643 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a268b276f30143818e61a95f0a39d40d [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:14.476043 14646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:14.455577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57352 (local address 127.10.191.254:37823)
0620 23:11:14.456330 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:14.456341 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:14.456385 (+ 44us) server_negotiation.cc:407] Connection header received
0620 23:11:14.458099 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:14.458121 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:14.458484 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:14.458830 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:14.460650 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:14.462090 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:14.463538 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:14.464193 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:14.464411 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:14.468644 (+ 4233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:14.468655 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:14.468668 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:14.468708 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:14.471812 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:14.472590 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:14.472596 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:14.472602 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:14.472764 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:14.473164 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:14.473167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:14.473169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:14.473652 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:14.473878 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:14.474285 (+ 407us) server_negotiation.cc:299] Negotiation successful
0620 23:11:14.474746 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":363,"threads_started":1}
I20260620 23:11:14.478101 14643 raft_consensus.cc:504] T 00000000000000000000000000000000 P a268b276f30143818e61a95f0a39d40d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a268b276f30143818e61a95f0a39d40d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 37823 } }
I20260620 23:11:14.478652 14643 leader_election.cc:302] T 00000000000000000000000000000000 P a268b276f30143818e61a95f0a39d40d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a268b276f30143818e61a95f0a39d40d; no voters:
I20260620 23:11:14.479242 14643 leader_election.cc:288] T 00000000000000000000000000000000 P a268b276f30143818e61a95f0a39d40d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:11:14.479384 14648 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a268b276f30143818e61a95f0a39d40d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:11:14.480070 14648 raft_consensus.cc:686] T 00000000000000000000000000000000 P a268b276f30143818e61a95f0a39d40d [term 1 LEADER]: Becoming Leader. State: Replica: a268b276f30143818e61a95f0a39d40d, State: Running, Role: LEADER
I20260620 23:11:14.480590 14648 consensus_queue.cc:237] T 00000000000000000000000000000000 P a268b276f30143818e61a95f0a39d40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a268b276f30143818e61a95f0a39d40d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 37823 } }
I20260620 23:11:14.481278 14643 sys_catalog.cc:564] T 00000000000000000000000000000000 P a268b276f30143818e61a95f0a39d40d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:11:14.483069 14649 sys_catalog.cc:455] T 00000000000000000000000000000000 P a268b276f30143818e61a95f0a39d40d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a268b276f30143818e61a95f0a39d40d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a268b276f30143818e61a95f0a39d40d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 37823 } } }
I20260620 23:11:14.483417 14649 sys_catalog.cc:458] T 00000000000000000000000000000000 P a268b276f30143818e61a95f0a39d40d [sys.catalog]: This master's current role is: LEADER
I20260620 23:11:14.483572 14650 sys_catalog.cc:455] T 00000000000000000000000000000000 P a268b276f30143818e61a95f0a39d40d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a268b276f30143818e61a95f0a39d40d. Latest consensus state: current_term: 1 leader_uuid: "a268b276f30143818e61a95f0a39d40d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a268b276f30143818e61a95f0a39d40d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 37823 } } }
I20260620 23:11:14.484218 14650 sys_catalog.cc:458] T 00000000000000000000000000000000 P a268b276f30143818e61a95f0a39d40d [sys.catalog]: This master's current role is: LEADER
I20260620 23:11:14.487474 14653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:11:14.492177 14653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:11:14.500437 14653 catalog_manager.cc:1347] Generated new cluster ID: c951a3bc9ed04307a307a5130c6fc802
I20260620 23:11:14.500545 14653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:11:14.515610 14653 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:11:14.517488 14653 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:11:14.528973 14653 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a268b276f30143818e61a95f0a39d40d: Generated new TSK 0
I20260620 23:11:14.529767 14653 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.191.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab.
Entry for principal kudu/127.10.191.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab.
I20260620 23:11:14.586252 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.193:0
--local_ip_for_outbound_sockets=127.10.191.193
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:37823
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:43855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:11:14.905436 14670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:14.905699 14670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:14.905860 14670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:14.912978 14670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:14.913076 14670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:14.913244 14670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.193
I20260620 23:11:14.922544 14670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:43855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:37823
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.193
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:14.923898 14670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:14.925177 14670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:14.937486 14675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:14.937634 14678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:14.937456 14676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:14.938256 14670 server_base.cc:1034] running on GCE node
I20260620 23:11:14.938774 14670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:14.939651 14670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:14.940907 14670 hybrid_clock.cc:648] HybridClock initialized: now 1781997074940846 us; error 52 us; skew 500 ppm
I20260620 23:11:14.941226 14670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:11:14 dist-test-slave-jkbq krb5kdc[14563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997074, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:14.944244 14670 init.cc:377] Logged in from keytab as kudu/127.10.191.193@KRBTEST.COM (short username kudu)
I20260620 23:11:14.945662 14670 webserver.cc:460] Webserver started at http://127.10.191.193:44953/ using document root <none> and password file <none>
I20260620 23:11:14.946317 14670 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:14.946455 14670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:14.946754 14670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:11:14.949548 14670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-0/data/instance:
uuid: "f9b7636040cd440fa1698b98286a833d"
format_stamp: "Formatted at 2026-06-20 23:11:14 on dist-test-slave-jkbq"
server_key: "febca8832479db7041b31a53d2c9bb0f"
server_key_iv: "15057ed02e86958bbc929b35bcd91002"
server_key_version: "encryptionkey@0"
I20260620 23:11:14.950326 14670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance:
uuid: "f9b7636040cd440fa1698b98286a833d"
format_stamp: "Formatted at 2026-06-20 23:11:14 on dist-test-slave-jkbq"
server_key: "febca8832479db7041b31a53d2c9bb0f"
server_key_iv: "15057ed02e86958bbc929b35bcd91002"
server_key_version: "encryptionkey@0"
I20260620 23:11:14.955331 14670 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260620 23:11:14.958784 14685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:14.960119 14670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:11:14.960307 14670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-0/wal
uuid: "f9b7636040cd440fa1698b98286a833d"
format_stamp: "Formatted at 2026-06-20 23:11:14 on dist-test-slave-jkbq"
server_key: "febca8832479db7041b31a53d2c9bb0f"
server_key_iv: "15057ed02e86958bbc929b35bcd91002"
server_key_version: "encryptionkey@0"
I20260620 23:11:14.960508 14670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:14.978530 14670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:14.982697 14670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:14.983021 14670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:14.983913 14670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:11:14.985311 14670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:11:14.985410 14670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:14.985574 14670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:11:14.985646 14670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:15.018603 14670 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.193:38793
I20260620 23:11:15.018648 14797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.193:38793 every 8 connection(s)
I20260620 23:11:15.019923 14670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
I20260620 23:11:15.025655 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 14670
I20260620 23:11:15.025878 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance
I20260620 23:11:15.026679 11007 external_mini_cluster.cc:1428] Setting key d49682a90e53f15a6b993079f8e39125
Jun 20 23:11:15 dist-test-slave-jkbq krb5kdc[14563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997074, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:11:15.043066 14800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:15.022396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:44545 (local address 127.10.191.254:37823)
0620 23:11:15.022844 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:15.022848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:15.023750 (+ 902us) server_negotiation.cc:407] Connection header received
0620 23:11:15.025761 (+ 2011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:15.025767 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:15.025862 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:15.025992 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:15.027842 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:15.028915 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:15.031468 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:15.031911 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:15.032038 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:15.036230 (+ 4192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:15.036240 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:15.036248 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:15.036291 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:15.039784 (+ 3493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:15.040493 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:15.040502 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:15.040507 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:15.040656 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:15.041075 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:15.041082 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:15.041089 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:15.041470 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:15.041625 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:15.042292 (+ 667us) server_negotiation.cc:299] Negotiation successful
0620 23:11:15.042537 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":253,"threads_started":1}
I20260620 23:11:15.043779 14801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:15.022842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:37823 (local address 127.10.191.193:44545)
0620 23:11:15.023583 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:15.023625 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:15.025414 (+ 1789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:15.026268 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:15.026282 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:15.027057 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:15.027711 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:15.027730 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:15.029067 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:15.029073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:15.031305 (+ 2232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:15.031323 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:15.032151 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:15.032160 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:15.032493 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:15.033252 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:15.033287 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:15.035983 (+ 2696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:15.039917 (+ 3934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:15.039923 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:15.039936 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:15.040347 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:15.040772 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:15.040775 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:15.040777 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:15.040947 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:15.041591 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:15.041597 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:15.041910 (+ 313us) client_negotiation.cc:769] Sending connection context
0620 23:11:15.042133 (+ 223us) client_negotiation.cc:240] Negotiation successful
0620 23:11:15.042514 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":255,"threads_started":1}
I20260620 23:11:15.045364 14798 heartbeater.cc:347] Connected to a master server at 127.10.191.254:37823
I20260620 23:11:15.045706 14798 heartbeater.cc:464] Registering TS with master...
I20260620 23:11:15.046651 14798 heartbeater.cc:511] Master 127.10.191.254:37823 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.191.194@KRBTEST.COM; defaulting to no policy
I20260620 23:11:15.048795 14608 ts_manager.cc:194] Registered new tserver with Master: f9b7636040cd440fa1698b98286a833d (127.10.191.193:38793)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.194@KRBTEST.COM" created.
I20260620 23:11:15.050253 14608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.193@KRBTEST.COM'} at 127.10.191.193:44545
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab.
Entry for principal kudu/127.10.191.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab.
I20260620 23:11:15.073765 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.194:0
--local_ip_for_outbound_sockets=127.10.191.194
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:37823
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:43855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:11:15.387795 14804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:15.388072 14804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:15.388229 14804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:15.396304 14804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:15.396442 14804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:15.396621 14804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.194
I20260620 23:11:15.406481 14804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:43855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:37823
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.194
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:15.407805 14804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:15.409116 14804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:15.421962 14810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:15.422035 14812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:15.422035 14809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:15.423040 14804 server_base.cc:1034] running on GCE node
I20260620 23:11:15.423586 14804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:15.424386 14804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:15.425618 14804 hybrid_clock.cc:648] HybridClock initialized: now 1781997075425550 us; error 52 us; skew 500 ppm
I20260620 23:11:15.425959 14804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:11:15 dist-test-slave-jkbq krb5kdc[14563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997075, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:15.429003 14804 init.cc:377] Logged in from keytab as kudu/127.10.191.194@KRBTEST.COM (short username kudu)
I20260620 23:11:15.430310 14804 webserver.cc:460] Webserver started at http://127.10.191.194:42943/ using document root <none> and password file <none>
I20260620 23:11:15.431028 14804 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:15.431155 14804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:15.431470 14804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:11:15.434243 14804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-1/data/instance:
uuid: "34028885dd6c48e0ae6bd689355004d8"
format_stamp: "Formatted at 2026-06-20 23:11:15 on dist-test-slave-jkbq"
server_key: "4d5ee2ecfff51cc2efb342b9559e53c2"
server_key_iv: "3c071ee192bedaadbb59a4efa3c2fd6d"
server_key_version: "encryptionkey@0"
I20260620 23:11:15.435055 14804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance:
uuid: "34028885dd6c48e0ae6bd689355004d8"
format_stamp: "Formatted at 2026-06-20 23:11:15 on dist-test-slave-jkbq"
server_key: "4d5ee2ecfff51cc2efb342b9559e53c2"
server_key_iv: "3c071ee192bedaadbb59a4efa3c2fd6d"
server_key_version: "encryptionkey@0"
I20260620 23:11:15.439904 14804 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 23:11:15.443452 14819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:15.444818 14804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 23:11:15.444985 14804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-1/wal
uuid: "34028885dd6c48e0ae6bd689355004d8"
format_stamp: "Formatted at 2026-06-20 23:11:15 on dist-test-slave-jkbq"
server_key: "4d5ee2ecfff51cc2efb342b9559e53c2"
server_key_iv: "3c071ee192bedaadbb59a4efa3c2fd6d"
server_key_version: "encryptionkey@0"
I20260620 23:11:15.445195 14804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:15.472826 14804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:15.476640 14804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:15.476924 14804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:15.477867 14804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:11:15.479187 14804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:11:15.479285 14804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:15.479430 14804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:11:15.479502 14804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:15.511394 14804 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.194:36973
I20260620 23:11:15.511443 14931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.194:36973 every 8 connection(s)
I20260620 23:11:15.512639 14804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
I20260620 23:11:15.521584 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 14804
I20260620 23:11:15.521803 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance
I20260620 23:11:15.522212 11007 external_mini_cluster.cc:1428] Setting key 6774c8c6d5df36e8c59968937fb479e8
Jun 20 23:11:15 dist-test-slave-jkbq krb5kdc[14563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997075, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:11:15.532756 14800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:15.515192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:46257 (local address 127.10.191.254:37823)
0620 23:11:15.515367 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:15.515372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:15.516558 (+ 1186us) server_negotiation.cc:407] Connection header received
0620 23:11:15.518352 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:15.518357 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:15.518426 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:15.518539 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:15.520008 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:15.520737 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:15.522305 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:15.522942 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:15.523065 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:15.527033 (+ 3968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:15.527038 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:15.527042 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:15.527072 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:15.529775 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:15.530480 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:15.530489 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:15.530492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:15.530607 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:15.531104 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:15.531109 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:15.531112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:15.531374 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:15.531522 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:15.532200 (+ 678us) server_negotiation.cc:299] Negotiation successful
0620 23:11:15.532412 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:11:15.533903 14934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:15.515732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:37823 (local address 127.10.191.194:46257)
0620 23:11:15.516400 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:15.516443 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:15.518078 (+ 1635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:15.518722 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:15.518730 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:15.519299 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:15.519843 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:15.519860 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:15.520867 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:15.520878 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:15.522168 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:15.522179 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:15.523143 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:15.523148 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:15.523332 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:15.524023 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:15.524055 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:15.526767 (+ 2712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:15.529910 (+ 3143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:15.529920 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:15.529936 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:15.530343 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:15.530720 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:15.530726 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:15.530730 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:15.530987 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:15.531497 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:15.531504 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:15.531856 (+ 352us) client_negotiation.cc:769] Sending connection context
0620 23:11:15.532113 (+ 257us) client_negotiation.cc:240] Negotiation successful
0620 23:11:15.532665 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":278,"threads_started":1}
I20260620 23:11:15.535357 14932 heartbeater.cc:347] Connected to a master server at 127.10.191.254:37823
I20260620 23:11:15.535681 14932 heartbeater.cc:464] Registering TS with master...
I20260620 23:11:15.536586 14932 heartbeater.cc:511] Master 127.10.191.254:37823 requested a full tablet report, sending...
I20260620 23:11:15.538226 14608 ts_manager.cc:194] Registered new tserver with Master: 34028885dd6c48e0ae6bd689355004d8 (127.10.191.194:36973)
I20260620 23:11:15.539083 14608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.194@KRBTEST.COM'} at 127.10.191.194:46257
WARNING: no policy specified for kudu/127.10.191.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab.
Entry for principal kudu/127.10.191.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab.
I20260620 23:11:15.565912 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.195:0
--local_ip_for_outbound_sockets=127.10.191.195
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:37823
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:43855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:11:15.878593 14937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:15.878932 14937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:15.879107 14937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:15.886313 14937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:15.886454 14937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:15.886634 14937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.195
I20260620 23:11:15.896420 14937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:43855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:37823
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.195
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:15.897758 14937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:15.899148 14937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:15.912158 14942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:15.912273 14943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:15.912184 14945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:15.913232 14937 server_base.cc:1034] running on GCE node
I20260620 23:11:15.913808 14937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:15.914707 14937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:15.915967 14937 hybrid_clock.cc:648] HybridClock initialized: now 1781997075915895 us; error 66 us; skew 500 ppm
I20260620 23:11:15.916327 14937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:11:15 dist-test-slave-jkbq krb5kdc[14563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997075, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:15.919381 14937 init.cc:377] Logged in from keytab as kudu/127.10.191.195@KRBTEST.COM (short username kudu)
I20260620 23:11:15.920902 14937 webserver.cc:460] Webserver started at http://127.10.191.195:43405/ using document root <none> and password file <none>
I20260620 23:11:15.921662 14937 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:15.921790 14937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:15.922113 14937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:11:15.925002 14937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-2/data/instance:
uuid: "973c529454f545d6b7a9bb97444f6e9b"
format_stamp: "Formatted at 2026-06-20 23:11:15 on dist-test-slave-jkbq"
server_key: "ad7e591689235dc3c97268eecc39e573"
server_key_iv: "8e7ed2e597dc5d89b998062f84f534c1"
server_key_version: "encryptionkey@0"
I20260620 23:11:15.925771 14937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance:
uuid: "973c529454f545d6b7a9bb97444f6e9b"
format_stamp: "Formatted at 2026-06-20 23:11:15 on dist-test-slave-jkbq"
server_key: "ad7e591689235dc3c97268eecc39e573"
server_key_iv: "8e7ed2e597dc5d89b998062f84f534c1"
server_key_version: "encryptionkey@0"
I20260620 23:11:15.930632 14937 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 23:11:15.934059 14952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:15.935318 14937 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:11:15.935496 14937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-2/wal
uuid: "973c529454f545d6b7a9bb97444f6e9b"
format_stamp: "Formatted at 2026-06-20 23:11:15 on dist-test-slave-jkbq"
server_key: "ad7e591689235dc3c97268eecc39e573"
server_key_iv: "8e7ed2e597dc5d89b998062f84f534c1"
server_key_version: "encryptionkey@0"
I20260620 23:11:15.935686 14937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:15.965397 14937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:15.969434 14937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:15.969722 14937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:15.970642 14937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:11:15.971940 14937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:11:15.972043 14937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:15.972193 14937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:11:15.972296 14937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:16.005474 14937 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.195:42037
I20260620 23:11:16.005509 15064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.195:42037 every 8 connection(s)
I20260620 23:11:16.006781 14937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
I20260620 23:11:16.013048 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 14937
I20260620 23:11:16.013247 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance
I20260620 23:11:16.013666 11007 external_mini_cluster.cc:1428] Setting key 8754733ca30977e9e35842c4e613cf59
Jun 20 23:11:16 dist-test-slave-jkbq krb5kdc[14563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997075, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:11:16.026232 14800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:16.009148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:39507 (local address 127.10.191.254:37823)
0620 23:11:16.009319 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:16.009324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:16.010417 (+ 1093us) server_negotiation.cc:407] Connection header received
0620 23:11:16.012228 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:16.012233 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:16.012320 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:16.012415 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:16.014004 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:16.014929 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:16.016621 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:16.017072 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:16.017164 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:16.020755 (+ 3591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:16.020760 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:16.020763 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:16.020792 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:16.023445 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:16.024110 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:16.024118 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:16.024121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:16.024237 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:16.024638 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:16.024645 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:16.024648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:16.024913 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:16.025065 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:16.025674 (+ 609us) server_negotiation.cc:299] Negotiation successful
0620 23:11:16.025890 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:11:16.027223 15067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:16.009592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:37823 (local address 127.10.191.195:39507)
0620 23:11:16.010243 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:16.010279 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:16.011983 (+ 1704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:16.012617 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:16.012625 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:16.013186 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:16.013842 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:16.013860 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:16.015063 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:16.015067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:16.016494 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:16.016505 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:16.017249 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:16.017254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:16.017439 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:16.018057 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:16.018085 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:16.020556 (+ 2471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:16.023582 (+ 3026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:16.023590 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:16.023604 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:16.023994 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:16.024340 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:16.024346 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:16.024349 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:16.024531 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:16.025038 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:16.025045 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:16.025347 (+ 302us) client_negotiation.cc:769] Sending connection context
0620 23:11:16.025602 (+ 255us) client_negotiation.cc:240] Negotiation successful
0620 23:11:16.026062 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":270,"threads_started":1}
I20260620 23:11:16.028627 15065 heartbeater.cc:347] Connected to a master server at 127.10.191.254:37823
I20260620 23:11:16.028921 15065 heartbeater.cc:464] Registering TS with master...
I20260620 23:11:16.029539 15065 heartbeater.cc:511] Master 127.10.191.254:37823 requested a full tablet report, sending...
I20260620 23:11:16.030840 14608 ts_manager.cc:194] Registered new tserver with Master: 973c529454f545d6b7a9bb97444f6e9b (127.10.191.195:42037)
I20260620 23:11:16.031677 14608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.195@KRBTEST.COM'} at 127.10.191.195:39507
I20260620 23:11:16.040910 11007 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:11:16.053313 14798 heartbeater.cc:503] Master 127.10.191.254:37823 was elected leader, sending a full tablet report...
Jun 20 23:11:16 dist-test-slave-jkbq krb5kdc[14563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 23:11:16 dist-test-slave-jkbq krb5kdc[14563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.193@KRBTEST.COM
I20260620 23:11:16.087608 15072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:16.073030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34072 (local address 127.10.191.193:38793)
0620 23:11:16.074018 (+ 988us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:16.074025 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:16.074052 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:11:16.074110 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:16.074117 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:16.074328 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:16.074454 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:16.075834 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:16.076851 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:16.078011 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:16.078556 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:16.078656 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:16.082485 (+ 3829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:16.082495 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:16.082509 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:16.082542 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:16.085381 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:16.085956 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:16.085962 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:16.085965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:16.086081 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:16.086371 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:16.086374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:16.086376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:16.086677 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:16.086958 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:16.087149 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 23:11:16.087330 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":827,"thread_start_us":687,"threads_started":1}
Jun 20 23:11:16 dist-test-slave-jkbq krb5kdc[14563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:11:16.104794 14800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:16.091727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57368 (local address 127.10.191.254:37823)
0620 23:11:16.091952 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:16.091959 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:16.092182 (+ 223us) server_negotiation.cc:407] Connection header received
0620 23:11:16.092347 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:16.092351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:16.092413 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:16.092506 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:16.093659 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:16.094353 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:16.095625 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:16.096028 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:16.096153 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:16.099733 (+ 3580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:16.099739 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:16.099742 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:16.099772 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:16.102227 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:16.102902 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:16.102912 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:16.102914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:16.103070 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:16.103424 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:16.103431 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:16.103434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:16.103739 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:16.103983 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:16.104267 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 23:11:16.104471 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 23:11:16.105337 14608 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57368
I20260620 23:11:16.106314 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 14670
I20260620 23:11:16.118515 15072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:16.107202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34080 (local address 127.10.191.193:38793)
0620 23:11:16.107409 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:16.107416 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:16.107616 (+ 200us) server_negotiation.cc:407] Connection header received
0620 23:11:16.107802 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:16.107806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:16.107863 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:16.107960 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:16.109054 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:16.109722 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:16.111037 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:16.111470 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:16.111599 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:16.113685 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:16.113691 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:16.113695 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:16.113732 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:16.116140 (+ 2408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:16.116732 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:16.116739 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:16.116743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:16.116865 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:16.117214 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:16.117220 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:16.117223 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:16.117467 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:16.117700 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:16.118001 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 23:11:16.118205 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:11:16.119180 14793 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:11:16.200917 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 14670
I20260620 23:11:16.217787 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 14804
Jun 20 23:11:16 dist-test-slave-jkbq krb5kdc[14563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.194@KRBTEST.COM
I20260620 23:11:16.233184 15077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:16.218690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43316 (local address 127.10.191.194:36973)
0620 23:11:16.219270 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:16.219277 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:16.219303 (+ 26us) server_negotiation.cc:407] Connection header received
0620 23:11:16.219373 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:16.219380 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:16.219605 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:16.219764 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:16.220821 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:16.221818 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:16.223195 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:16.223771 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:16.223887 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:16.227755 (+ 3868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:16.227766 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:16.227782 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:16.227816 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:16.230676 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:16.231392 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:16.231399 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:16.231401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:16.231514 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:16.231841 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:16.231845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:16.231847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:16.232163 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:16.232379 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:16.232589 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 23:11:16.232819 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":299,"threads_started":1}
I20260620 23:11:16.234460 14927 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:11:16.320406 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 14804
I20260620 23:11:16.335837 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 14937
Jun 20 23:11:16 dist-test-slave-jkbq krb5kdc[14563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.195@KRBTEST.COM
I20260620 23:11:16.352144 15079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:16.336897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47994 (local address 127.10.191.195:42037)
0620 23:11:16.337514 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:16.337521 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:16.337551 (+ 30us) server_negotiation.cc:407] Connection header received
0620 23:11:16.337609 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:16.337616 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:16.337837 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:16.337972 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:16.339284 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:16.340410 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:16.341637 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:16.342210 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:16.342313 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:16.346421 (+ 4108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:16.346433 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:16.346447 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:16.346478 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:16.349389 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:16.350090 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:16.350098 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:16.350100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:16.350223 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:16.350557 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:16.350560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:16.350562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:16.350968 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:16.351223 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:16.351548 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 23:11:16.351739 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":330,"threads_started":1}
I20260620 23:11:16.353529 15060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:11:16.438516 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 14937
I20260620 23:11:16.458200 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 14576
I20260620 23:11:16.459422 14638 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:11:16.559146 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 14576
2026-06-20T23:11:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3826 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:11:16 dist-test-slave-jkbq krb5kdc[15084](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:11:16 dist-test-slave-jkbq krb5kdc[15084](info): set up 2 sockets
Jun 20 23:11:16 dist-test-slave-jkbq krb5kdc[15084](info): commencing operation
krb5kdc: starting...
W20260620 23:11:17.643298 11007 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Jun 20 23:11:17 dist-test-slave-jkbq krb5kdc[15084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:11:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:11:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.191.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
I20260620 23:11:17.828205 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:42407
--webserver_interface=127.10.191.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:45169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:42407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:11:18.135830 15097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:18.136103 15097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:18.136279 15097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:18.143271 15097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:11:18.143364 15097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:18.143468 15097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:18.143559 15097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:11:18.143640 15097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:11:18.152988 15097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:45169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:42407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:42407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:18.154256 15097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:18.155542 15097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:18.166534 15102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:18.166635 15103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:18.166601 15105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:18.167721 15097 server_base.cc:1034] running on GCE node
I20260620 23:11:18.168382 15097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:18.169602 15097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:18.170800 15097 hybrid_clock.cc:648] HybridClock initialized: now 1781997078170764 us; error 30 us; skew 500 ppm
I20260620 23:11:18.171132 15097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:11:18 dist-test-slave-jkbq krb5kdc[15084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997078, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:18.173964 15097 init.cc:377] Logged in from keytab as kudu/127.10.191.254@KRBTEST.COM (short username kudu)
I20260620 23:11:18.175160 15097 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:18.175266 15097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:18.175592 15097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:11:18.178365 15097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "00d1195cd7074637a45b048e6981f893"
format_stamp: "Formatted at 2026-06-20 23:11:18 on dist-test-slave-jkbq"
server_key: "0d5ce6903d35df9df30bb24e625f9707"
server_key_iv: "bbe38fd05eb07c54ddf4933227aca4e0"
server_key_version: "encryptionkey@0"
I20260620 23:11:18.179169 15097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "00d1195cd7074637a45b048e6981f893"
format_stamp: "Formatted at 2026-06-20 23:11:18 on dist-test-slave-jkbq"
server_key: "0d5ce6903d35df9df30bb24e625f9707"
server_key_iv: "bbe38fd05eb07c54ddf4933227aca4e0"
server_key_version: "encryptionkey@0"
I20260620 23:11:18.183941 15097 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 23:11:18.187136 15111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:18.188388 15097 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:11:18.188565 15097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "00d1195cd7074637a45b048e6981f893"
format_stamp: "Formatted at 2026-06-20 23:11:18 on dist-test-slave-jkbq"
server_key: "0d5ce6903d35df9df30bb24e625f9707"
server_key_iv: "bbe38fd05eb07c54ddf4933227aca4e0"
server_key_version: "encryptionkey@0"
I20260620 23:11:18.188771 15097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:18.240258 15097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:18.244194 15097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:18.244472 15097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:18.262816 15097 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:42407
I20260620 23:11:18.262837 15162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:42407 every 8 connection(s)
I20260620 23:11:18.263762 15097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:11:18.265106 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 15097
I20260620 23:11:18.265329 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:11:18.266093 11007 external_mini_cluster.cc:1428] Setting key 2776ccba171ff5b7d92198644875bd2d
I20260620 23:11:18.268486 15163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 23:11:18 dist-test-slave-jkbq krb5kdc[15084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:11:18.282006 15163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00d1195cd7074637a45b048e6981f893: Bootstrap starting.
I20260620 23:11:18.286856 15163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00d1195cd7074637a45b048e6981f893: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:18.288209 15163 log.cc:824] T 00000000000000000000000000000000 P 00d1195cd7074637a45b048e6981f893: Log is configured to *not* fsync() on all Append() calls
I20260620 23:11:18.291700 15163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00d1195cd7074637a45b048e6981f893: No bootstrap required, opened a new log
I20260620 23:11:18.292340 15166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:18.268173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51596 (local address 127.10.191.254:42407)
0620 23:11:18.268941 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:18.268953 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:18.268991 (+ 38us) server_negotiation.cc:407] Connection header received
0620 23:11:18.270544 (+ 1553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:18.270562 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:18.271003 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:18.271329 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:18.273180 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:18.274502 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:18.276336 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:18.276986 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:18.277106 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:18.282223 (+ 5117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:18.282234 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:18.282258 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:18.282320 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:18.286687 (+ 4367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:18.287644 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:18.287652 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:18.287664 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:18.287932 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:18.288299 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:18.288304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:18.288308 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:18.289050 (+ 742us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:18.289358 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:18.290560 (+ 1202us) server_negotiation.cc:299] Negotiation successful
0620 23:11:18.290996 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":368,"threads_started":1}
I20260620 23:11:18.300364 15163 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00d1195cd7074637a45b048e6981f893 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d1195cd7074637a45b048e6981f893" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 42407 } }
I20260620 23:11:18.300679 15163 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00d1195cd7074637a45b048e6981f893 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:18.300742 15163 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00d1195cd7074637a45b048e6981f893 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00d1195cd7074637a45b048e6981f893, State: Initialized, Role: FOLLOWER
I20260620 23:11:18.301306 15163 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00d1195cd7074637a45b048e6981f893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d1195cd7074637a45b048e6981f893" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 42407 } }
I20260620 23:11:18.301505 15163 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00d1195cd7074637a45b048e6981f893 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:11:18.301596 15163 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00d1195cd7074637a45b048e6981f893 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:11:18.301715 15163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00d1195cd7074637a45b048e6981f893 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:18.304082 15163 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00d1195cd7074637a45b048e6981f893 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d1195cd7074637a45b048e6981f893" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 42407 } }
I20260620 23:11:18.304497 15163 leader_election.cc:302] T 00000000000000000000000000000000 P 00d1195cd7074637a45b048e6981f893 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00d1195cd7074637a45b048e6981f893; no voters:
I20260620 23:11:18.305059 15163 leader_election.cc:288] T 00000000000000000000000000000000 P 00d1195cd7074637a45b048e6981f893 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:11:18.305138 15168 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00d1195cd7074637a45b048e6981f893 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:11:18.305817 15168 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00d1195cd7074637a45b048e6981f893 [term 1 LEADER]: Becoming Leader. State: Replica: 00d1195cd7074637a45b048e6981f893, State: Running, Role: LEADER
I20260620 23:11:18.306356 15168 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00d1195cd7074637a45b048e6981f893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d1195cd7074637a45b048e6981f893" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 42407 } }
I20260620 23:11:18.307191 15163 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00d1195cd7074637a45b048e6981f893 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:11:18.309101 15169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00d1195cd7074637a45b048e6981f893 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00d1195cd7074637a45b048e6981f893" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d1195cd7074637a45b048e6981f893" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 42407 } } }
I20260620 23:11:18.309398 15169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00d1195cd7074637a45b048e6981f893 [sys.catalog]: This master's current role is: LEADER
I20260620 23:11:18.310199 15174 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:11:18.310061 15170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00d1195cd7074637a45b048e6981f893 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00d1195cd7074637a45b048e6981f893. Latest consensus state: current_term: 1 leader_uuid: "00d1195cd7074637a45b048e6981f893" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d1195cd7074637a45b048e6981f893" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 42407 } } }
I20260620 23:11:18.311241 15170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00d1195cd7074637a45b048e6981f893 [sys.catalog]: This master's current role is: LEADER
I20260620 23:11:18.314779 15174 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:11:18.323807 15174 catalog_manager.cc:1347] Generated new cluster ID: 34a80ed12576433bbe4e08d4b01900c5
I20260620 23:11:18.323913 15174 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:11:18.336216 15174 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:11:18.337925 15174 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:11:18.348155 15174 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00d1195cd7074637a45b048e6981f893: Generated new TSK 0
I20260620 23:11:18.349066 15174 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.191.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab.
Entry for principal kudu/127.10.191.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab.
I20260620 23:11:18.400283 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.193:0
--local_ip_for_outbound_sockets=127.10.191.193
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:42407
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:45169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:11:18.718992 15189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:18.719241 15189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:18.719406 15189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:18.726300 15189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:18.726399 15189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:18.726567 15189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.193
I20260620 23:11:18.735895 15189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:45169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:42407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.193
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:18.737174 15189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:18.738435 15189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:18.751184 15194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:18.751128 15195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:18.751636 15197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:18.751989 15189 server_base.cc:1034] running on GCE node
I20260620 23:11:18.752487 15189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:18.753317 15189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:18.754510 15189 hybrid_clock.cc:648] HybridClock initialized: now 1781997078754448 us; error 52 us; skew 500 ppm
I20260620 23:11:18.754818 15189 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:11:18 dist-test-slave-jkbq krb5kdc[15084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997078, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:18.757853 15189 init.cc:377] Logged in from keytab as kudu/127.10.191.193@KRBTEST.COM (short username kudu)
I20260620 23:11:18.759006 15189 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:18.759133 15189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:18.759436 15189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:11:18.762341 15189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-0/data/instance:
uuid: "2ee5a9ba024b4263adadd9edecfda186"
format_stamp: "Formatted at 2026-06-20 23:11:18 on dist-test-slave-jkbq"
server_key: "c80d0520aab40bd493b25623cf2622b5"
server_key_iv: "a6daa0e3f21e250e5da48c6d7d0883f2"
server_key_version: "encryptionkey@0"
I20260620 23:11:18.763196 15189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance:
uuid: "2ee5a9ba024b4263adadd9edecfda186"
format_stamp: "Formatted at 2026-06-20 23:11:18 on dist-test-slave-jkbq"
server_key: "c80d0520aab40bd493b25623cf2622b5"
server_key_iv: "a6daa0e3f21e250e5da48c6d7d0883f2"
server_key_version: "encryptionkey@0"
I20260620 23:11:18.767948 15189 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 23:11:18.771202 15203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:18.772476 15189 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:11:18.772653 15189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-0/wal
uuid: "2ee5a9ba024b4263adadd9edecfda186"
format_stamp: "Formatted at 2026-06-20 23:11:18 on dist-test-slave-jkbq"
server_key: "c80d0520aab40bd493b25623cf2622b5"
server_key_iv: "a6daa0e3f21e250e5da48c6d7d0883f2"
server_key_version: "encryptionkey@0"
I20260620 23:11:18.772886 15189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:18.784598 15189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:18.788705 15189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:18.788971 15189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:18.789893 15189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:11:18.791311 15189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:11:18.791441 15189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:18.791569 15189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:11:18.791666 15189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:18.825245 15189 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.193:44121
I20260620 23:11:18.825299 15315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.193:44121 every 8 connection(s)
I20260620 23:11:18.826076 15189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
I20260620 23:11:18.828673 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 15189
I20260620 23:11:18.828929 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance
I20260620 23:11:18.829526 11007 external_mini_cluster.cc:1428] Setting key e2272f0a809e21feb9987c09e50c089f
Jun 20 23:11:18 dist-test-slave-jkbq krb5kdc[15084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997078, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:11:18.847584 15318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:18.828512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:43277 (local address 127.10.191.254:42407)
0620 23:11:18.829017 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:18.829023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:18.830935 (+ 1912us) server_negotiation.cc:407] Connection header received
0620 23:11:18.832076 (+ 1141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:18.832084 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:18.832173 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:18.832285 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:18.833670 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:18.834830 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:18.836410 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:18.836812 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:18.836947 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:18.840831 (+ 3884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:18.840840 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:18.840848 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:18.840891 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:18.844317 (+ 3426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:18.845107 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:18.845113 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:18.845119 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:18.845258 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:18.845649 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:18.845666 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:18.845671 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:18.846067 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:18.846229 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:18.846892 (+ 663us) server_negotiation.cc:299] Negotiation successful
0620 23:11:18.847131 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":248,"threads_started":1}
I20260620 23:11:18.848482 15319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:18.829075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:42407 (local address 127.10.191.193:43277)
0620 23:11:18.829876 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:18.829927 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:18.831823 (+ 1896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:18.832473 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:18.832481 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:18.832986 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:18.833512 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:18.833527 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:18.835006 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:18.835011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:18.836253 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:18.836264 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:18.837032 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:18.837039 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:18.837321 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:18.837994 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:18.838031 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:18.840603 (+ 2572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:18.844463 (+ 3860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:18.844470 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:18.844483 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:18.844987 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:18.845378 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:18.845381 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:18.845383 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:18.845539 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:18.846197 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:18.846203 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:18.846515 (+ 312us) client_negotiation.cc:769] Sending connection context
0620 23:11:18.846736 (+ 221us) client_negotiation.cc:240] Negotiation successful
0620 23:11:18.847249 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":337,"threads_started":1}
WARNING: no policy specified for kudu/127.10.191.194@KRBTEST.COM; defaulting to no policy
I20260620 23:11:18.849993 15316 heartbeater.cc:347] Connected to a master server at 127.10.191.254:42407
I20260620 23:11:18.850255 15316 heartbeater.cc:464] Registering TS with master...
I20260620 23:11:18.850989 15316 heartbeater.cc:511] Master 127.10.191.254:42407 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.194@KRBTEST.COM" created.
I20260620 23:11:18.852934 15128 ts_manager.cc:194] Registered new tserver with Master: 2ee5a9ba024b4263adadd9edecfda186 (127.10.191.193:44121)
I20260620 23:11:18.854327 15128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.193@KRBTEST.COM'} at 127.10.191.193:43277
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab.
Entry for principal kudu/127.10.191.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab.
I20260620 23:11:18.874436 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.194:0
--local_ip_for_outbound_sockets=127.10.191.194
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:42407
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:45169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:11:19.187719 15322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:19.187983 15322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:19.188144 15322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:19.195359 15322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:19.195463 15322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:19.195678 15322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.194
I20260620 23:11:19.205013 15322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:45169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:42407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.194
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:19.206283 15322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:19.207825 15322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:19.220330 15330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:19.220367 15327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:19.220504 15328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:19.221556 15322 server_base.cc:1034] running on GCE node
I20260620 23:11:19.222168 15322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:19.223151 15322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:19.224360 15322 hybrid_clock.cc:648] HybridClock initialized: now 1781997079224324 us; error 35 us; skew 500 ppm
I20260620 23:11:19.224704 15322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:11:19 dist-test-slave-jkbq krb5kdc[15084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997079, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:19.232758 15322 init.cc:377] Logged in from keytab as kudu/127.10.191.194@KRBTEST.COM (short username kudu)
I20260620 23:11:19.233974 15322 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:19.234088 15322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:19.234411 15322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:11:19.237310 15322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-1/data/instance:
uuid: "a9b6bdc1b82e4d39b444452a5e0bc855"
format_stamp: "Formatted at 2026-06-20 23:11:19 on dist-test-slave-jkbq"
server_key: "25acb5724414c701fa993ee6648fbd52"
server_key_iv: "72887dca823cebeca6f6818d979c127a"
server_key_version: "encryptionkey@0"
I20260620 23:11:19.238094 15322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance:
uuid: "a9b6bdc1b82e4d39b444452a5e0bc855"
format_stamp: "Formatted at 2026-06-20 23:11:19 on dist-test-slave-jkbq"
server_key: "25acb5724414c701fa993ee6648fbd52"
server_key_iv: "72887dca823cebeca6f6818d979c127a"
server_key_version: "encryptionkey@0"
I20260620 23:11:19.243038 15322 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 23:11:19.246268 15336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:19.247562 15322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:11:19.247743 15322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-1/wal
uuid: "a9b6bdc1b82e4d39b444452a5e0bc855"
format_stamp: "Formatted at 2026-06-20 23:11:19 on dist-test-slave-jkbq"
server_key: "25acb5724414c701fa993ee6648fbd52"
server_key_iv: "72887dca823cebeca6f6818d979c127a"
server_key_version: "encryptionkey@0"
I20260620 23:11:19.247952 15322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:19.270025 15322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:19.274181 15322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:19.274454 15322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:19.275568 15322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:11:19.277020 15322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:11:19.277130 15322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:19.277257 15322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:11:19.277347 15322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:19.310483 15322 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.194:46511
I20260620 23:11:19.310530 15448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.194:46511 every 8 connection(s)
I20260620 23:11:19.311650 15322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
I20260620 23:11:19.313485 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 15322
I20260620 23:11:19.313675 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance
I20260620 23:11:19.314352 11007 external_mini_cluster.cc:1428] Setting key 0f869f586e3eed2bd0b314cc4ea59778
Jun 20 23:11:19 dist-test-slave-jkbq krb5kdc[15084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997079, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.191.195@KRBTEST.COM; defaulting to no policy
I20260620 23:11:19.342293 15318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:19.314960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:41363 (local address 127.10.191.254:42407)
0620 23:11:19.315167 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:19.315173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:19.319217 (+ 4044us) server_negotiation.cc:407] Connection header received
0620 23:11:19.322484 (+ 3267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:19.322487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:19.322549 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:19.322635 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:19.325254 (+ 2619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:19.326352 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:19.329092 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:19.329832 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:19.329943 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:19.334786 (+ 4843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:19.334790 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:19.334794 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:19.334824 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:19.338576 (+ 3752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:19.339599 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:19.339607 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:19.339613 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:19.339775 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:19.340320 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:19.340324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:19.340328 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:19.340700 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:19.340876 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:19.341780 (+ 904us) server_negotiation.cc:299] Negotiation successful
0620 23:11:19.341946 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 23:11:19.343883 15451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:19.315562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:42407 (local address 127.10.191.194:41363)
0620 23:11:19.319032 (+ 3470us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:19.319093 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:19.322177 (+ 3084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:19.322973 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:19.322986 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:19.323783 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:19.324698 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:19.324724 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:19.326508 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:19.326517 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:19.328922 (+ 2405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:19.328939 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:19.330286 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:19.330295 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:19.330619 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:19.331471 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:19.331504 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:19.334565 (+ 3061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:19.338736 (+ 4171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:19.338746 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:19.338766 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:19.339456 (+ 690us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:19.339891 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:19.339897 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:19.339902 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:19.340144 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:19.340847 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:19.340855 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:19.341371 (+ 516us) client_negotiation.cc:769] Sending connection context
0620 23:11:19.341682 (+ 311us) client_negotiation.cc:240] Negotiation successful
0620 23:11:19.342289 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3132,"thread_start_us":2916,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.195@KRBTEST.COM" created.
I20260620 23:11:19.346958 15449 heartbeater.cc:347] Connected to a master server at 127.10.191.254:42407
I20260620 23:11:19.347275 15449 heartbeater.cc:464] Registering TS with master...
I20260620 23:11:19.348006 15449 heartbeater.cc:511] Master 127.10.191.254:42407 requested a full tablet report, sending...
I20260620 23:11:19.349542 15128 ts_manager.cc:194] Registered new tserver with Master: a9b6bdc1b82e4d39b444452a5e0bc855 (127.10.191.194:46511)
I20260620 23:11:19.350528 15128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.194@KRBTEST.COM'} at 127.10.191.194:41363
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab.
Entry for principal kudu/127.10.191.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab.
I20260620 23:11:19.369737 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.195:0
--local_ip_for_outbound_sockets=127.10.191.195
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:42407
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:45169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:11:19.686272 15454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:19.686539 15454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:19.686733 15454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:19.694197 15454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:19.694339 15454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:19.694532 15454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.195
I20260620 23:11:19.704336 15454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:45169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:42407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.195
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:19.705623 15454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:19.706954 15454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:19.719965 15460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:19.719938 15462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:19.720044 15459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:19.720855 15454 server_base.cc:1034] running on GCE node
I20260620 23:11:19.721429 15454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:19.722321 15454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:19.723538 15454 hybrid_clock.cc:648] HybridClock initialized: now 1781997079723497 us; error 53 us; skew 500 ppm
I20260620 23:11:19.723860 15454 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:11:19 dist-test-slave-jkbq krb5kdc[15084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997079, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:19.726783 15454 init.cc:377] Logged in from keytab as kudu/127.10.191.195@KRBTEST.COM (short username kudu)
I20260620 23:11:19.727942 15454 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:19.728157 15454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:19.728507 15454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:11:19.731415 15454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-2/data/instance:
uuid: "458b016762334d029e15b6a695076692"
format_stamp: "Formatted at 2026-06-20 23:11:19 on dist-test-slave-jkbq"
server_key: "f8831702a85884ac0da0198928cf452d"
server_key_iv: "28fbb10cd4f63faab854a40bb78dc6c2"
server_key_version: "encryptionkey@0"
I20260620 23:11:19.732196 15454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance:
uuid: "458b016762334d029e15b6a695076692"
format_stamp: "Formatted at 2026-06-20 23:11:19 on dist-test-slave-jkbq"
server_key: "f8831702a85884ac0da0198928cf452d"
server_key_iv: "28fbb10cd4f63faab854a40bb78dc6c2"
server_key_version: "encryptionkey@0"
I20260620 23:11:19.737053 15454 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 23:11:19.740356 15468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:19.741650 15454 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:11:19.741837 15454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-2/wal
uuid: "458b016762334d029e15b6a695076692"
format_stamp: "Formatted at 2026-06-20 23:11:19 on dist-test-slave-jkbq"
server_key: "f8831702a85884ac0da0198928cf452d"
server_key_iv: "28fbb10cd4f63faab854a40bb78dc6c2"
server_key_version: "encryptionkey@0"
I20260620 23:11:19.742055 15454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:19.757936 15454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:19.761814 15454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:19.762084 15454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:19.763093 15454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:11:19.764530 15454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:11:19.764657 15454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:19.764792 15454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:11:19.764880 15454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:19.797858 15454 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.195:33839
I20260620 23:11:19.797945 15580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.195:33839 every 8 connection(s)
I20260620 23:11:19.798763 15454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
I20260620 23:11:19.809062 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 15454
I20260620 23:11:19.809245 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance
I20260620 23:11:19.809767 11007 external_mini_cluster.cc:1428] Setting key d2a93d288272ae86278a33a302e56f07
Jun 20 23:11:19 dist-test-slave-jkbq krb5kdc[15084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997079, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:11:19.818835 15318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:19.801445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:51345 (local address 127.10.191.254:42407)
0620 23:11:19.801620 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:19.801625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:19.802889 (+ 1264us) server_negotiation.cc:407] Connection header received
0620 23:11:19.804692 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:19.804699 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:19.804822 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:19.804941 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:19.806418 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:19.807270 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:19.808894 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:19.809424 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:19.809529 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:19.813634 (+ 4105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:19.813639 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:19.813642 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:19.813672 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:19.816370 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:19.817023 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:19.817027 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:19.817029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:19.817134 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:19.817474 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:19.817477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:19.817478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:19.817745 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:19.817852 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:19.818432 (+ 580us) server_negotiation.cc:299] Negotiation successful
0620 23:11:19.818575 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 23:11:19.819890 15583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:19.801982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:42407 (local address 127.10.191.195:51345)
0620 23:11:19.802699 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:19.802742 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:19.804409 (+ 1667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:19.805167 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:19.805175 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:19.805683 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:19.806238 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:19.806259 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:19.807442 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:19.807454 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:19.808742 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:19.808758 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:19.809646 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:19.809653 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:19.809944 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:19.810752 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:19.810783 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:19.813450 (+ 2667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:19.816498 (+ 3048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:19.816505 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:19.816517 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:19.816926 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:19.817221 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:19.817224 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:19.817226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:19.817386 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:19.817848 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:19.817854 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:19.818137 (+ 283us) client_negotiation.cc:769] Sending connection context
0620 23:11:19.818374 (+ 237us) client_negotiation.cc:240] Negotiation successful
0620 23:11:19.818726 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":290,"threads_started":1}
I20260620 23:11:19.821177 15581 heartbeater.cc:347] Connected to a master server at 127.10.191.254:42407
I20260620 23:11:19.821480 15581 heartbeater.cc:464] Registering TS with master...
I20260620 23:11:19.822100 15581 heartbeater.cc:511] Master 127.10.191.254:42407 requested a full tablet report, sending...
I20260620 23:11:19.823534 15128 ts_manager.cc:194] Registered new tserver with Master: 458b016762334d029e15b6a695076692 (127.10.191.195:33839)
I20260620 23:11:19.824671 15128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.195@KRBTEST.COM'} at 127.10.191.195:51345
I20260620 23:11:19.825264 11007 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:11:19.841390 15318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:19.829967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51600 (local address 127.10.191.254:42407)
0620 23:11:19.830148 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:19.830152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:19.830656 (+ 504us) server_negotiation.cc:407] Connection header received
0620 23:11:19.830790 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:19.830793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:19.830847 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:19.830985 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:19.832251 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:19.832940 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:19.834126 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:19.834536 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:19.834624 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:19.836847 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:19.836852 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:19.836855 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:19.836887 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:19.839288 (+ 2401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:19.839872 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:19.839876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:19.839878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:19.839992 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:19.840284 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:19.840287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:19.840289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:19.840576 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:19.840740 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:19.841025 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 23:11:19.841171 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 23:11:19.845413 15128 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:11:19.852246 15128 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:11:19.857270 15316 heartbeater.cc:503] Master 127.10.191.254:42407 was elected leader, sending a full tablet report...
I20260620 23:11:19.877553 15595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:19.864927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:44121 (local address 127.0.0.1:53722)
0620 23:11:19.866701 (+ 1774us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:19.866722 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:19.866844 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:19.869505 (+ 2661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:19.869509 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:19.869528 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:11:19.870017 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:19.870029 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:19.872706 (+ 2677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:19.872711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:19.874781 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:19.874797 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:19.876577 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:19.876584 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:19.876834 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:19.876873 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 23:11:19.876958 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 23:11:19.877125 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1623,"thread_start_us":3054,"threads_started":1}
I20260620 23:11:19.878194 15593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:19.863902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.195:33839 (local address 127.0.0.1:48254)
0620 23:11:19.865734 (+ 1832us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:19.865777 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:19.865984 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:19.869304 (+ 3320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:19.869311 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:19.869344 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:11:19.870136 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:19.870147 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:19.873086 (+ 2939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:19.873094 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:19.875303 (+ 2209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:19.875319 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:19.877320 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:19.877327 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:19.877513 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:19.877558 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 23:11:19.877646 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 23:11:19.877786 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1525,"mutex_wait_us":322,"thread_start_us":1369,"threads_started":1}
I20260620 23:11:19.878387 15596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:19.865472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53722 (local address 127.10.191.193:44121)
0620 23:11:19.868957 (+ 3485us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:19.868965 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:19.868987 (+ 22us) server_negotiation.cc:407] Connection header received
0620 23:11:19.869088 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:19.869098 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:19.869299 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:19.870253 (+ 954us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:11:19.870467 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:19.872002 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:19.875000 (+ 2998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:19.876301 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:19.877661 (+ 1360us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:19.877827 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:19.877903 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 23:11:19.878063 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3347,"thread_start_us":3205,"threads_started":1}
I20260620 23:11:19.878974 15594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:19.864185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48254 (local address 127.10.191.195:33839)
0620 23:11:19.868639 (+ 4454us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:19.868651 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:19.868688 (+ 37us) server_negotiation.cc:407] Connection header received
0620 23:11:19.868791 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:19.868799 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:19.869056 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:19.869841 (+ 785us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:11:19.870355 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:19.872940 (+ 2585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:19.875481 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:19.877102 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:19.877292 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:19.877462 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:19.878504 (+ 1042us) server_negotiation.cc:299] Negotiation successful
0620 23:11:19.878662 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4281,"thread_start_us":4126,"threads_started":1}
I20260620 23:11:19.880524 15597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:19.866128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.194:46511 (local address 127.0.0.1:53058)
0620 23:11:19.872461 (+ 6333us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:19.872477 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:19.872571 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:19.873298 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:19.873306 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:19.873330 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:11:19.873829 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:19.873841 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:19.875756 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:19.875761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:19.877818 (+ 2057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:19.877832 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:19.879709 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:19.879715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:19.879869 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:19.879911 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 23:11:19.879997 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 23:11:19.880122 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6207,"mutex_wait_us":2,"thread_start_us":6085,"threads_started":1}
I20260620 23:11:19.881245 15598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:19.866619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53058 (local address 127.10.191.194:46511)
0620 23:11:19.872774 (+ 6155us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:19.872780 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:19.872801 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:11:19.872872 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:19.872876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:19.873066 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:19.873269 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:11:19.874041 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:19.875592 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:19.878001 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:19.879513 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:19.879709 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:19.879859 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:19.880721 (+ 862us) server_negotiation.cc:299] Negotiation successful
0620 23:11:19.880894 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6014,"thread_start_us":5874,"threads_started":1}
I20260620 23:11:19.882385 15516 tablet_service.cc:1467] Processing CreateTablet for tablet c72f45485a0f4451902e537a70d923e2 (DEFAULT_TABLE table=test-table [id=79d01588fc7b40a8b43120565318cf28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:11:19.882385 15251 tablet_service.cc:1467] Processing CreateTablet for tablet c72f45485a0f4451902e537a70d923e2 (DEFAULT_TABLE table=test-table [id=79d01588fc7b40a8b43120565318cf28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:11:19.883839 15251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c72f45485a0f4451902e537a70d923e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:11:19.883858 15516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c72f45485a0f4451902e537a70d923e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:11:19.884052 15384 tablet_service.cc:1467] Processing CreateTablet for tablet c72f45485a0f4451902e537a70d923e2 (DEFAULT_TABLE table=test-table [id=79d01588fc7b40a8b43120565318cf28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:11:19.885080 15384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c72f45485a0f4451902e537a70d923e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:11:19.896376 15599 tablet_bootstrap.cc:492] T c72f45485a0f4451902e537a70d923e2 P a9b6bdc1b82e4d39b444452a5e0bc855: Bootstrap starting.
I20260620 23:11:19.899554 15600 tablet_bootstrap.cc:492] T c72f45485a0f4451902e537a70d923e2 P 2ee5a9ba024b4263adadd9edecfda186: Bootstrap starting.
I20260620 23:11:19.899609 15599 tablet_bootstrap.cc:654] T c72f45485a0f4451902e537a70d923e2 P a9b6bdc1b82e4d39b444452a5e0bc855: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:19.900998 15599 log.cc:824] T c72f45485a0f4451902e537a70d923e2 P a9b6bdc1b82e4d39b444452a5e0bc855: Log is configured to *not* fsync() on all Append() calls
I20260620 23:11:19.903365 15600 tablet_bootstrap.cc:654] T c72f45485a0f4451902e537a70d923e2 P 2ee5a9ba024b4263adadd9edecfda186: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:19.903697 15601 tablet_bootstrap.cc:492] T c72f45485a0f4451902e537a70d923e2 P 458b016762334d029e15b6a695076692: Bootstrap starting.
I20260620 23:11:19.904582 15600 log.cc:824] T c72f45485a0f4451902e537a70d923e2 P 2ee5a9ba024b4263adadd9edecfda186: Log is configured to *not* fsync() on all Append() calls
I20260620 23:11:19.905915 15599 tablet_bootstrap.cc:492] T c72f45485a0f4451902e537a70d923e2 P a9b6bdc1b82e4d39b444452a5e0bc855: No bootstrap required, opened a new log
I20260620 23:11:19.906183 15599 ts_tablet_manager.cc:1397] T c72f45485a0f4451902e537a70d923e2 P a9b6bdc1b82e4d39b444452a5e0bc855: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260620 23:11:19.907038 15601 tablet_bootstrap.cc:654] T c72f45485a0f4451902e537a70d923e2 P 458b016762334d029e15b6a695076692: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:19.907544 15600 tablet_bootstrap.cc:492] T c72f45485a0f4451902e537a70d923e2 P 2ee5a9ba024b4263adadd9edecfda186: No bootstrap required, opened a new log
I20260620 23:11:19.907809 15600 ts_tablet_manager.cc:1397] T c72f45485a0f4451902e537a70d923e2 P 2ee5a9ba024b4263adadd9edecfda186: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 23:11:19.908193 15601 log.cc:824] T c72f45485a0f4451902e537a70d923e2 P 458b016762334d029e15b6a695076692: Log is configured to *not* fsync() on all Append() calls
I20260620 23:11:19.914003 15599 raft_consensus.cc:348] T c72f45485a0f4451902e537a70d923e2 P a9b6bdc1b82e4d39b444452a5e0bc855 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458b016762334d029e15b6a695076692" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 33839 } } peers { permanent_uuid: "a9b6bdc1b82e4d39b444452a5e0bc855" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 46511 } } peers { permanent_uuid: "2ee5a9ba024b4263adadd9edecfda186" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 44121 } }
I20260620 23:11:19.914458 15599 raft_consensus.cc:374] T c72f45485a0f4451902e537a70d923e2 P a9b6bdc1b82e4d39b444452a5e0bc855 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:19.914561 15599 raft_consensus.cc:729] T c72f45485a0f4451902e537a70d923e2 P a9b6bdc1b82e4d39b444452a5e0bc855 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9b6bdc1b82e4d39b444452a5e0bc855, State: Initialized, Role: FOLLOWER
I20260620 23:11:19.915266 15599 consensus_queue.cc:260] T c72f45485a0f4451902e537a70d923e2 P a9b6bdc1b82e4d39b444452a5e0bc855 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458b016762334d029e15b6a695076692" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 33839 } } peers { permanent_uuid: "a9b6bdc1b82e4d39b444452a5e0bc855" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 46511 } } peers { permanent_uuid: "2ee5a9ba024b4263adadd9edecfda186" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 44121 } }
I20260620 23:11:19.916359 15449 heartbeater.cc:503] Master 127.10.191.254:42407 was elected leader, sending a full tablet report...
I20260620 23:11:19.917057 15599 ts_tablet_manager.cc:1428] T c72f45485a0f4451902e537a70d923e2 P a9b6bdc1b82e4d39b444452a5e0bc855: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260620 23:11:19.917105 15601 tablet_bootstrap.cc:492] T c72f45485a0f4451902e537a70d923e2 P 458b016762334d029e15b6a695076692: No bootstrap required, opened a new log
I20260620 23:11:19.917359 15601 ts_tablet_manager.cc:1397] T c72f45485a0f4451902e537a70d923e2 P 458b016762334d029e15b6a695076692: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.006s
I20260620 23:11:19.920933 15600 raft_consensus.cc:348] T c72f45485a0f4451902e537a70d923e2 P 2ee5a9ba024b4263adadd9edecfda186 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458b016762334d029e15b6a695076692" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 33839 } } peers { permanent_uuid: "a9b6bdc1b82e4d39b444452a5e0bc855" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 46511 } } peers { permanent_uuid: "2ee5a9ba024b4263adadd9edecfda186" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 44121 } }
I20260620 23:11:19.921355 15600 raft_consensus.cc:374] T c72f45485a0f4451902e537a70d923e2 P 2ee5a9ba024b4263adadd9edecfda186 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:19.921464 15600 raft_consensus.cc:729] T c72f45485a0f4451902e537a70d923e2 P 2ee5a9ba024b4263adadd9edecfda186 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ee5a9ba024b4263adadd9edecfda186, State: Initialized, Role: FOLLOWER
I20260620 23:11:19.922086 15600 consensus_queue.cc:260] T c72f45485a0f4451902e537a70d923e2 P 2ee5a9ba024b4263adadd9edecfda186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458b016762334d029e15b6a695076692" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 33839 } } peers { permanent_uuid: "a9b6bdc1b82e4d39b444452a5e0bc855" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 46511 } } peers { permanent_uuid: "2ee5a9ba024b4263adadd9edecfda186" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 44121 } }
I20260620 23:11:19.923650 15600 ts_tablet_manager.cc:1428] T c72f45485a0f4451902e537a70d923e2 P 2ee5a9ba024b4263adadd9edecfda186: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260620 23:11:19.925416 15601 raft_consensus.cc:348] T c72f45485a0f4451902e537a70d923e2 P 458b016762334d029e15b6a695076692 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458b016762334d029e15b6a695076692" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 33839 } } peers { permanent_uuid: "a9b6bdc1b82e4d39b444452a5e0bc855" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 46511 } } peers { permanent_uuid: "2ee5a9ba024b4263adadd9edecfda186" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 44121 } }
I20260620 23:11:19.925813 15601 raft_consensus.cc:374] T c72f45485a0f4451902e537a70d923e2 P 458b016762334d029e15b6a695076692 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:19.925921 15601 raft_consensus.cc:729] T c72f45485a0f4451902e537a70d923e2 P 458b016762334d029e15b6a695076692 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 458b016762334d029e15b6a695076692, State: Initialized, Role: FOLLOWER
I20260620 23:11:19.926503 15601 consensus_queue.cc:260] T c72f45485a0f4451902e537a70d923e2 P 458b016762334d029e15b6a695076692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458b016762334d029e15b6a695076692" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 33839 } } peers { permanent_uuid: "a9b6bdc1b82e4d39b444452a5e0bc855" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 46511 } } peers { permanent_uuid: "2ee5a9ba024b4263adadd9edecfda186" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 44121 } }
I20260620 23:11:19.927538 15581 heartbeater.cc:503] Master 127.10.191.254:42407 was elected leader, sending a full tablet report...
I20260620 23:11:19.928001 15601 ts_tablet_manager.cc:1428] T c72f45485a0f4451902e537a70d923e2 P 458b016762334d029e15b6a695076692: Time spent starting tablet: real 0.011s user 0.005s sys 0.006s
W20260620 23:11:20.050319 15582 tablet.cc:2369] T c72f45485a0f4451902e537a70d923e2 P 458b016762334d029e15b6a695076692: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:11:20.063848 15450 tablet.cc:2369] T c72f45485a0f4451902e537a70d923e2 P a9b6bdc1b82e4d39b444452a5e0bc855: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:11:20.078293 15317 tablet.cc:2369] T c72f45485a0f4451902e537a70d923e2 P 2ee5a9ba024b4263adadd9edecfda186: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:11:20.088824 15606 raft_consensus.cc:482] T c72f45485a0f4451902e537a70d923e2 P 2ee5a9ba024b4263adadd9edecfda186 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:11:20.089079 15606 raft_consensus.cc:504] T c72f45485a0f4451902e537a70d923e2 P 2ee5a9ba024b4263adadd9edecfda186 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458b016762334d029e15b6a695076692" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 33839 } } peers { permanent_uuid: "a9b6bdc1b82e4d39b444452a5e0bc855" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 46511 } } peers { permanent_uuid: "2ee5a9ba024b4263adadd9edecfda186" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 44121 } }
I20260620 23:11:20.090672 15606 leader_election.cc:288] T c72f45485a0f4451902e537a70d923e2 P 2ee5a9ba024b4263adadd9edecfda186 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 458b016762334d029e15b6a695076692 (127.10.191.195:33839), a9b6bdc1b82e4d39b444452a5e0bc855 (127.10.191.194:46511)
I20260620 23:11:20.099253 15598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:20.092044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:35963 (local address 127.10.191.194:46511)
0620 23:11:20.092262 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:20.092268 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:20.092531 (+ 263us) server_negotiation.cc:407] Connection header received
0620 23:11:20.092735 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:20.092739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:20.092807 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:20.092884 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:11:20.093465 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:20.094459 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:20.097005 (+ 2546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:20.098250 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:20.098400 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:20.098470 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:20.098833 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 23:11:20.098995 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 23:11:20.099349 15609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:20.091815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.194:46511 (local address 127.10.191.193:35963)
0620 23:11:20.092437 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:20.092456 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:20.092563 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:20.092938 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:20.092943 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:20.092995 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:11:20.093327 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:20.093335 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:20.094578 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:20.094582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:20.096821 (+ 2239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:20.096841 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:20.098469 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:20.098475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:20.098647 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:20.098684 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 23:11:20.098810 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 23:11:20.098965 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":44,"thread_start_us":341,"threads_started":1}
I20260620 23:11:20.099439 15594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:20.091186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:40419 (local address 127.10.191.195:33839)
0620 23:11:20.091396 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:20.091404 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:20.091981 (+ 577us) server_negotiation.cc:407] Connection header received
0620 23:11:20.092158 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:20.092167 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:20.092260 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:20.092327 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:11:20.093162 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:20.094269 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:20.097009 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:20.098253 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:20.098405 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:20.098470 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:20.099077 (+ 607us) server_negotiation.cc:299] Negotiation successful
0620 23:11:20.099203 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 23:11:20.100129 15608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:20.091101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.195:33839 (local address 127.10.191.193:40419)
0620 23:11:20.091770 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:20.091792 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:20.091956 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:20.092544 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:20.092550 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:20.092569 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:11:20.092938 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:20.092948 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:20.094394 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:20.094397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:20.096840 (+ 2443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:20.096849 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:20.098674 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:20.098684 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:20.098891 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:20.098927 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 23:11:20.099610 (+ 683us) client_negotiation.cc:240] Negotiation successful
0620 23:11:20.099757 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"mutex_wait_us":49,"thread_start_us":323,"threads_started":1}
I20260620 23:11:20.100661 15404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c72f45485a0f4451902e537a70d923e2" candidate_uuid: "2ee5a9ba024b4263adadd9edecfda186" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9b6bdc1b82e4d39b444452a5e0bc855" is_pre_election: true
I20260620 23:11:20.101128 15536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c72f45485a0f4451902e537a70d923e2" candidate_uuid: "2ee5a9ba024b4263adadd9edecfda186" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "458b016762334d029e15b6a695076692" is_pre_election: true
I20260620 23:11:20.101285 15404 raft_consensus.cc:2454] T c72f45485a0f4451902e537a70d923e2 P a9b6bdc1b82e4d39b444452a5e0bc855 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ee5a9ba024b4263adadd9edecfda186 in term 0.
I20260620 23:11:20.101644 15536 raft_consensus.cc:2454] T c72f45485a0f4451902e537a70d923e2 P 458b016762334d029e15b6a695076692 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ee5a9ba024b4263adadd9edecfda186 in term 0.
I20260620 23:11:20.101980 15204 leader_election.cc:302] T c72f45485a0f4451902e537a70d923e2 P 2ee5a9ba024b4263adadd9edecfda186 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ee5a9ba024b4263adadd9edecfda186, a9b6bdc1b82e4d39b444452a5e0bc855; no voters:
I20260620 23:11:20.102350 15606 raft_consensus.cc:2781] T c72f45485a0f4451902e537a70d923e2 P 2ee5a9ba024b4263adadd9edecfda186 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:11:20.102510 15606 raft_consensus.cc:482] T c72f45485a0f4451902e537a70d923e2 P 2ee5a9ba024b4263adadd9edecfda186 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:11:20.102660 15606 raft_consensus.cc:3037] T c72f45485a0f4451902e537a70d923e2 P 2ee5a9ba024b4263adadd9edecfda186 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:20.104820 15606 raft_consensus.cc:504] T c72f45485a0f4451902e537a70d923e2 P 2ee5a9ba024b4263adadd9edecfda186 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458b016762334d029e15b6a695076692" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 33839 } } peers { permanent_uuid: "a9b6bdc1b82e4d39b444452a5e0bc855" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 46511 } } peers { permanent_uuid: "2ee5a9ba024b4263adadd9edecfda186" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 44121 } }
I20260620 23:11:20.105358 15606 leader_election.cc:288] T c72f45485a0f4451902e537a70d923e2 P 2ee5a9ba024b4263adadd9edecfda186 [CANDIDATE]: Term 1 election: Requested vote from peers 458b016762334d029e15b6a695076692 (127.10.191.195:33839), a9b6bdc1b82e4d39b444452a5e0bc855 (127.10.191.194:46511)
I20260620 23:11:20.105775 15536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c72f45485a0f4451902e537a70d923e2" candidate_uuid: "2ee5a9ba024b4263adadd9edecfda186" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "458b016762334d029e15b6a695076692"
I20260620 23:11:20.105866 15404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c72f45485a0f4451902e537a70d923e2" candidate_uuid: "2ee5a9ba024b4263adadd9edecfda186" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9b6bdc1b82e4d39b444452a5e0bc855"
I20260620 23:11:20.105948 15536 raft_consensus.cc:3037] T c72f45485a0f4451902e537a70d923e2 P 458b016762334d029e15b6a695076692 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:20.106096 15404 raft_consensus.cc:3037] T c72f45485a0f4451902e537a70d923e2 P a9b6bdc1b82e4d39b444452a5e0bc855 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:20.108255 15536 raft_consensus.cc:2454] T c72f45485a0f4451902e537a70d923e2 P 458b016762334d029e15b6a695076692 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ee5a9ba024b4263adadd9edecfda186 in term 1.
I20260620 23:11:20.108434 15404 raft_consensus.cc:2454] T c72f45485a0f4451902e537a70d923e2 P a9b6bdc1b82e4d39b444452a5e0bc855 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ee5a9ba024b4263adadd9edecfda186 in term 1.
I20260620 23:11:20.108625 15204 leader_election.cc:302] T c72f45485a0f4451902e537a70d923e2 P 2ee5a9ba024b4263adadd9edecfda186 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ee5a9ba024b4263adadd9edecfda186, 458b016762334d029e15b6a695076692; no voters:
I20260620 23:11:20.108909 15606 raft_consensus.cc:2781] T c72f45485a0f4451902e537a70d923e2 P 2ee5a9ba024b4263adadd9edecfda186 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:11:20.109319 15606 raft_consensus.cc:686] T c72f45485a0f4451902e537a70d923e2 P 2ee5a9ba024b4263adadd9edecfda186 [term 1 LEADER]: Becoming Leader. State: Replica: 2ee5a9ba024b4263adadd9edecfda186, State: Running, Role: LEADER
I20260620 23:11:20.109790 15606 consensus_queue.cc:237] T c72f45485a0f4451902e537a70d923e2 P 2ee5a9ba024b4263adadd9edecfda186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458b016762334d029e15b6a695076692" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 33839 } } peers { permanent_uuid: "a9b6bdc1b82e4d39b444452a5e0bc855" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 46511 } } peers { permanent_uuid: "2ee5a9ba024b4263adadd9edecfda186" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 44121 } }
I20260620 23:11:20.114075 15128 catalog_manager.cc:5495] T c72f45485a0f4451902e537a70d923e2 P 2ee5a9ba024b4263adadd9edecfda186 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ee5a9ba024b4263adadd9edecfda186 (127.10.191.193). New cstate: current_term: 1 leader_uuid: "2ee5a9ba024b4263adadd9edecfda186" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458b016762334d029e15b6a695076692" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 33839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9b6bdc1b82e4d39b444452a5e0bc855" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 46511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ee5a9ba024b4263adadd9edecfda186" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 44121 } health_report { overall_health: HEALTHY } } }
I20260620 23:11:20.167806 15596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:20.162479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53732 (local address 127.10.191.193:44121)
0620 23:11:20.162713 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:20.162722 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:20.162852 (+ 130us) server_negotiation.cc:407] Connection header received
0620 23:11:20.163048 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:20.163052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:20.163124 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:20.163247 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:11:20.163542 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:20.164214 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:20.165652 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:20.166041 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:20.166129 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:20.166522 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:11:20.167154 (+ 632us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:11:20.167325 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:20.167419 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 23:11:20.167500 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 23:11:20.175477 15404 raft_consensus.cc:1261] T c72f45485a0f4451902e537a70d923e2 P a9b6bdc1b82e4d39b444452a5e0bc855 [term 1 FOLLOWER]: Refusing update from remote peer 2ee5a9ba024b4263adadd9edecfda186: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:11:20.175510 15536 raft_consensus.cc:1261] T c72f45485a0f4451902e537a70d923e2 P 458b016762334d029e15b6a695076692 [term 1 FOLLOWER]: Refusing update from remote peer 2ee5a9ba024b4263adadd9edecfda186: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:11:20.176501 15610 consensus_queue.cc:1035] T c72f45485a0f4451902e537a70d923e2 P 2ee5a9ba024b4263adadd9edecfda186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9b6bdc1b82e4d39b444452a5e0bc855" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 46511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:11:20.176877 15606 consensus_queue.cc:1035] T c72f45485a0f4451902e537a70d923e2 P 2ee5a9ba024b4263adadd9edecfda186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "458b016762334d029e15b6a695076692" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 33839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:11:20.186408 15613 mvcc.cc:205] Tried to move back new op lower bound from 7299060040389480448 to 7299060040133083136. Current Snapshot: MvccSnapshot[applied={T|T < 7299060040389480448}]
I20260620 23:11:20.186967 15615 mvcc.cc:205] Tried to move back new op lower bound from 7299060040389480448 to 7299060040133083136. Current Snapshot: MvccSnapshot[applied={T|T < 7299060040389480448}]
I20260620 23:11:20.203465 15127 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:11:20.203727 15127 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:11:20.207343 15127 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 00d1195cd7074637a45b048e6981f893: Sending DeleteTablet for 3 replicas of tablet c72f45485a0f4451902e537a70d923e2
I20260620 23:11:20.208411 15516 tablet_service.cc:1514] Processing DeleteTablet for tablet c72f45485a0f4451902e537a70d923e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:11:20 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:48254
I20260620 23:11:20.208550 15384 tablet_service.cc:1514] Processing DeleteTablet for tablet c72f45485a0f4451902e537a70d923e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:11:20 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:53058
I20260620 23:11:20.208912 15251 tablet_service.cc:1514] Processing DeleteTablet for tablet c72f45485a0f4451902e537a70d923e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:11:20 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:53722
I20260620 23:11:20.210659 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 15189
I20260620 23:11:20.238200 15623 tablet_replica.cc:330] T c72f45485a0f4451902e537a70d923e2 P 458b016762334d029e15b6a695076692: stopping tablet replica
I20260620 23:11:20.238523 15625 tablet_replica.cc:330] T c72f45485a0f4451902e537a70d923e2 P 2ee5a9ba024b4263adadd9edecfda186: stopping tablet replica
I20260620 23:11:20.238718 15623 raft_consensus.cc:2229] T c72f45485a0f4451902e537a70d923e2 P 458b016762334d029e15b6a695076692 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:11:20.239149 15625 raft_consensus.cc:2229] T c72f45485a0f4451902e537a70d923e2 P 2ee5a9ba024b4263adadd9edecfda186 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:11:20.239295 15623 raft_consensus.cc:2258] T c72f45485a0f4451902e537a70d923e2 P 458b016762334d029e15b6a695076692 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:11:20.239948 15625 raft_consensus.cc:2258] T c72f45485a0f4451902e537a70d923e2 P 2ee5a9ba024b4263adadd9edecfda186 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:11:20.240839 15624 tablet_replica.cc:330] T c72f45485a0f4451902e537a70d923e2 P a9b6bdc1b82e4d39b444452a5e0bc855: stopping tablet replica
I20260620 23:11:20.241259 15624 raft_consensus.cc:2229] T c72f45485a0f4451902e537a70d923e2 P a9b6bdc1b82e4d39b444452a5e0bc855 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:11:20.241807 15624 raft_consensus.cc:2258] T c72f45485a0f4451902e537a70d923e2 P a9b6bdc1b82e4d39b444452a5e0bc855 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:11:20.242233 15625 ts_tablet_manager.cc:1905] T c72f45485a0f4451902e537a70d923e2 P 2ee5a9ba024b4263adadd9edecfda186: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:11:20.244077 15624 ts_tablet_manager.cc:1905] T c72f45485a0f4451902e537a70d923e2 P a9b6bdc1b82e4d39b444452a5e0bc855: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:11:20.245417 15623 ts_tablet_manager.cc:1905] T c72f45485a0f4451902e537a70d923e2 P 458b016762334d029e15b6a695076692: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:11:20.251370 15624 ts_tablet_manager.cc:1918] T c72f45485a0f4451902e537a70d923e2 P a9b6bdc1b82e4d39b444452a5e0bc855: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:11:20.251525 15624 log.cc:1196] T c72f45485a0f4451902e537a70d923e2 P a9b6bdc1b82e4d39b444452a5e0bc855: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-1/wal/wals/c72f45485a0f4451902e537a70d923e2
I20260620 23:11:20.252064 15624 ts_tablet_manager.cc:1939] T c72f45485a0f4451902e537a70d923e2 P a9b6bdc1b82e4d39b444452a5e0bc855: Deleting consensus metadata
I20260620 23:11:20.252631 15623 ts_tablet_manager.cc:1918] T c72f45485a0f4451902e537a70d923e2 P 458b016762334d029e15b6a695076692: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:11:20.252867 15623 log.cc:1196] T c72f45485a0f4451902e537a70d923e2 P 458b016762334d029e15b6a695076692: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-2/wal/wals/c72f45485a0f4451902e537a70d923e2
I20260620 23:11:20.253425 15623 ts_tablet_manager.cc:1939] T c72f45485a0f4451902e537a70d923e2 P 458b016762334d029e15b6a695076692: Deleting consensus metadata
I20260620 23:11:20.253912 15112 catalog_manager.cc:4841] TS a9b6bdc1b82e4d39b444452a5e0bc855 (127.10.191.194:46511): tablet c72f45485a0f4451902e537a70d923e2 (table test-table [id=79d01588fc7b40a8b43120565318cf28]) successfully deleted
I20260620 23:11:20.255224 15112 catalog_manager.cc:4841] TS 458b016762334d029e15b6a695076692 (127.10.191.195:33839): tablet c72f45485a0f4451902e537a70d923e2 (table test-table [id=79d01588fc7b40a8b43120565318cf28]) successfully deleted
Jun 20 23:11:20 dist-test-slave-jkbq krb5kdc[15084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.193@KRBTEST.COM
I20260620 23:11:20.256800 15625 ts_tablet_manager.cc:1918] T c72f45485a0f4451902e537a70d923e2 P 2ee5a9ba024b4263adadd9edecfda186: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:11:20.256975 15625 log.cc:1196] T c72f45485a0f4451902e537a70d923e2 P 2ee5a9ba024b4263adadd9edecfda186: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781997042893763-11007-0/minicluster-data/ts-0/wal/wals/c72f45485a0f4451902e537a70d923e2
I20260620 23:11:20.257530 15625 ts_tablet_manager.cc:1939] T c72f45485a0f4451902e537a70d923e2 P 2ee5a9ba024b4263adadd9edecfda186: Deleting consensus metadata
I20260620 23:11:20.259263 15114 catalog_manager.cc:4841] TS 2ee5a9ba024b4263adadd9edecfda186 (127.10.191.193:44121): tablet c72f45485a0f4451902e537a70d923e2 (table test-table [id=79d01588fc7b40a8b43120565318cf28]) successfully deleted
I20260620 23:11:20.264186 15596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:20.212298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53736 (local address 127.10.191.193:44121)
0620 23:11:20.212521 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:20.212530 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:20.237132 (+ 24602us) server_negotiation.cc:407] Connection header received
0620 23:11:20.239754 (+ 2622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:20.239762 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:20.239876 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:20.239978 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:20.243638 (+ 3660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:20.244843 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:20.251472 (+ 6629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:20.252171 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:20.252291 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:20.257321 (+ 5030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:20.257332 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:20.257345 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:20.257378 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:20.261110 (+ 3732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:20.261780 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:20.261795 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:20.261801 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:20.261987 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:20.262462 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:20.262468 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:20.262472 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:20.262965 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:20.263240 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:20.263544 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 23:11:20.263778 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 23:11:20.271632 15311 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:11:20.376039 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 15189
I20260620 23:11:20.394524 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 15322
Jun 20 23:11:20 dist-test-slave-jkbq krb5kdc[15084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.194@KRBTEST.COM
I20260620 23:11:20.408972 15598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:20.395320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53062 (local address 127.10.191.194:46511)
0620 23:11:20.395491 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:20.395495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:20.395512 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:11:20.395631 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:20.395635 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:20.395694 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:20.395777 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:20.396935 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:20.397670 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:20.399123 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:20.399486 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:20.399582 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:20.403544 (+ 3962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:20.403559 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:20.403574 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:20.403605 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:20.406219 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:20.406854 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:20.406999 (+ 145us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:20.407002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:20.407123 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:20.407481 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:20.407485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:20.407487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:20.407971 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:20.408197 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:20.408495 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 23:11:20.408721 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 23:11:20.409624 15444 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:11:20.498137 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 15322
I20260620 23:11:20.514518 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 15454
Jun 20 23:11:20 dist-test-slave-jkbq krb5kdc[15084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.195@KRBTEST.COM
I20260620 23:11:20.530031 15594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:20.515337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48258 (local address 127.10.191.195:33839)
0620 23:11:20.515513 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:20.515519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:20.515588 (+ 69us) server_negotiation.cc:407] Connection header received
0620 23:11:20.515802 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:20.515808 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:20.515889 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:20.516006 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:20.517236 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:20.517956 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:20.519347 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:20.519728 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:20.519875 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:20.524642 (+ 4767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:20.524657 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:20.524674 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:20.524706 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:20.527388 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:20.528008 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:20.528017 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:20.528026 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:20.528160 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:20.528553 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:20.528562 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:20.528564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:20.528894 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:20.529183 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:20.529491 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 23:11:20.529719 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 23:11:20.530663 15576 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:11:20.617416 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 15454
I20260620 23:11:20.635223 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 15097
I20260620 23:11:20.636377 15158 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:11:20.741972 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 15097
2026-06-20T23:11:20Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4183 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T23:11:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:11:20Z Disabled control of system clock
I20260620 23:11:20.806535 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:44225
--webserver_interface=127.10.191.254
--webserver_port=0
--builtin_ntp_servers=127.10.191.212:45265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:44225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 23:11:21.117069 15634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:21.117352 15634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:21.117455 15634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:21.124534 15634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:11:21.124639 15634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:21.124683 15634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:21.124738 15634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:11:21.124781 15634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:11:21.133460 15634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:45265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:44225
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:44225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:21.134665 15634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:21.136336 15634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:21.147549 15642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:21.147606 15639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:21.147596 15640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:21.148301 15634 server_base.cc:1034] running on GCE node
I20260620 23:11:21.149106 15634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:21.150534 15634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:21.151760 15634 hybrid_clock.cc:648] HybridClock initialized: now 1781997081151723 us; error 53 us; skew 500 ppm
I20260620 23:11:21.152097 15634 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:11:21.154409 15634 webserver.cc:460] Webserver started at http://127.10.191.254:35739/ using document root <none> and password file <none>
I20260620 23:11:21.155172 15634 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:21.155298 15634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:21.155613 15634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:11:21.158334 15634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "5fa53d42498c4c3498190660e8b43f44"
format_stamp: "Formatted at 2026-06-20 23:11:21 on dist-test-slave-jkbq"
server_key: "0c882c9528e2c86caf8dd0214ac4bd8c"
server_key_iv: "fc2cf133270e5a62a02a1c70138c84f9"
server_key_version: "encryptionkey@0"
I20260620 23:11:21.159188 15634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "5fa53d42498c4c3498190660e8b43f44"
format_stamp: "Formatted at 2026-06-20 23:11:21 on dist-test-slave-jkbq"
server_key: "0c882c9528e2c86caf8dd0214ac4bd8c"
server_key_iv: "fc2cf133270e5a62a02a1c70138c84f9"
server_key_version: "encryptionkey@0"
I20260620 23:11:21.164069 15634 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 23:11:21.167347 15648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:21.168592 15634 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:11:21.168797 15634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "5fa53d42498c4c3498190660e8b43f44"
format_stamp: "Formatted at 2026-06-20 23:11:21 on dist-test-slave-jkbq"
server_key: "0c882c9528e2c86caf8dd0214ac4bd8c"
server_key_iv: "fc2cf133270e5a62a02a1c70138c84f9"
server_key_version: "encryptionkey@0"
I20260620 23:11:21.169013 15634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:21.174516 15634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:21.175299 15634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:21.175585 15634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:21.193090 15634 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:44225
I20260620 23:11:21.193135 15699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:44225 every 8 connection(s)
I20260620 23:11:21.194382 15634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:11:21.198071 15700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:11:21.203440 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 15634
I20260620 23:11:21.203644 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:11:21.204276 11007 external_mini_cluster.cc:1428] Setting key 26a206bf02c8e24685a7fa0b60ee97a6
I20260620 23:11:21.210111 15700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fa53d42498c4c3498190660e8b43f44: Bootstrap starting.
I20260620 23:11:21.214561 15700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fa53d42498c4c3498190660e8b43f44: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:21.215469 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:21.206501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53380 (local address 127.10.191.254:44225)
0620 23:11:21.208102 (+ 1601us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:21.208120 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:21.208184 (+ 64us) server_negotiation.cc:407] Connection header received
0620 23:11:21.210856 (+ 2672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:21.210913 (+ 57us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:21.211493 (+ 580us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:21.211956 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:21.213424 (+ 1468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:21.213438 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:21.213463 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:11:21.213490 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:21.213568 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:11:21.213595 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:21.213832 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:21.213976 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 23:11:21.214268 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":953,"thread_start_us":1223,"threads_started":1}
I20260620 23:11:21.215854 15700 log.cc:824] T 00000000000000000000000000000000 P 5fa53d42498c4c3498190660e8b43f44: Log is configured to *not* fsync() on all Append() calls
I20260620 23:11:21.218803 15700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fa53d42498c4c3498190660e8b43f44: No bootstrap required, opened a new log
I20260620 23:11:21.226598 15700 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fa53d42498c4c3498190660e8b43f44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa53d42498c4c3498190660e8b43f44" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 44225 } }
I20260620 23:11:21.227058 15700 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fa53d42498c4c3498190660e8b43f44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:21.227170 15700 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fa53d42498c4c3498190660e8b43f44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fa53d42498c4c3498190660e8b43f44, State: Initialized, Role: FOLLOWER
I20260620 23:11:21.227734 15700 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fa53d42498c4c3498190660e8b43f44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa53d42498c4c3498190660e8b43f44" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 44225 } }
I20260620 23:11:21.227979 15700 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fa53d42498c4c3498190660e8b43f44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:11:21.228121 15700 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fa53d42498c4c3498190660e8b43f44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:11:21.228309 15700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fa53d42498c4c3498190660e8b43f44 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:21.230515 15700 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fa53d42498c4c3498190660e8b43f44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa53d42498c4c3498190660e8b43f44" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 44225 } }
I20260620 23:11:21.231129 15700 leader_election.cc:302] T 00000000000000000000000000000000 P 5fa53d42498c4c3498190660e8b43f44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fa53d42498c4c3498190660e8b43f44; no voters:
I20260620 23:11:21.231766 15700 leader_election.cc:288] T 00000000000000000000000000000000 P 5fa53d42498c4c3498190660e8b43f44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:11:21.231926 15705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fa53d42498c4c3498190660e8b43f44 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:11:21.233470 15705 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fa53d42498c4c3498190660e8b43f44 [term 1 LEADER]: Becoming Leader. State: Replica: 5fa53d42498c4c3498190660e8b43f44, State: Running, Role: LEADER
I20260620 23:11:21.233938 15700 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fa53d42498c4c3498190660e8b43f44 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:11:21.234061 15705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fa53d42498c4c3498190660e8b43f44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa53d42498c4c3498190660e8b43f44" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 44225 } }
I20260620 23:11:21.237210 15707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fa53d42498c4c3498190660e8b43f44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fa53d42498c4c3498190660e8b43f44. Latest consensus state: current_term: 1 leader_uuid: "5fa53d42498c4c3498190660e8b43f44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa53d42498c4c3498190660e8b43f44" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 44225 } } }
I20260620 23:11:21.237258 15706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fa53d42498c4c3498190660e8b43f44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fa53d42498c4c3498190660e8b43f44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa53d42498c4c3498190660e8b43f44" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 44225 } } }
I20260620 23:11:21.237639 15707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fa53d42498c4c3498190660e8b43f44 [sys.catalog]: This master's current role is: LEADER
I20260620 23:11:21.237663 15706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fa53d42498c4c3498190660e8b43f44 [sys.catalog]: This master's current role is: LEADER
I20260620 23:11:21.240309 15715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:11:21.244573 15715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:11:21.252838 15715 catalog_manager.cc:1347] Generated new cluster ID: ccf2c2fb1f9f48ab81cd732f1e7233b1
I20260620 23:11:21.252954 15715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:11:21.275708 15715 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:11:21.276149 15715 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:11:21.287556 15715 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fa53d42498c4c3498190660e8b43f44: Generated new TSK 0
I20260620 23:11:21.288409 15715 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:11:21.299790 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.193:0
--local_ip_for_outbound_sockets=127.10.191.193
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:44225
--builtin_ntp_servers=127.10.191.212:45265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 23:11:21.623936 15724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:21.624202 15724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:21.624367 15724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:21.631554 15724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:21.631652 15724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:21.631826 15724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.193
I20260620 23:11:21.641148 15724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:45265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:44225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.193
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:21.642443 15724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:21.644276 15724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:21.657133 15729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:21.657096 15732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:21.657182 15730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:21.658207 15724 server_base.cc:1034] running on GCE node
I20260620 23:11:21.658831 15724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:21.659802 15724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:21.661077 15724 hybrid_clock.cc:648] HybridClock initialized: now 1781997081661034 us; error 46 us; skew 500 ppm
I20260620 23:11:21.661423 15724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:11:21.667068 15724 webserver.cc:460] Webserver started at http://127.10.191.193:34685/ using document root <none> and password file <none>
I20260620 23:11:21.667773 15724 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:21.667909 15724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:21.668246 15724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:11:21.671113 15724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-0/data/instance:
uuid: "51aaf66d2a6446fdb2c96b03d03c4c33"
format_stamp: "Formatted at 2026-06-20 23:11:21 on dist-test-slave-jkbq"
server_key: "5cd1cc0d2a9174157ff095504b82ddbd"
server_key_iv: "f753c5685daf86efa395504764e5a888"
server_key_version: "encryptionkey@0"
I20260620 23:11:21.671936 15724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance:
uuid: "51aaf66d2a6446fdb2c96b03d03c4c33"
format_stamp: "Formatted at 2026-06-20 23:11:21 on dist-test-slave-jkbq"
server_key: "5cd1cc0d2a9174157ff095504b82ddbd"
server_key_iv: "f753c5685daf86efa395504764e5a888"
server_key_version: "encryptionkey@0"
I20260620 23:11:21.676923 15724 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 23:11:21.680263 15738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:21.681597 15724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260620 23:11:21.681725 15724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-0/wal
uuid: "51aaf66d2a6446fdb2c96b03d03c4c33"
format_stamp: "Formatted at 2026-06-20 23:11:21 on dist-test-slave-jkbq"
server_key: "5cd1cc0d2a9174157ff095504b82ddbd"
server_key_iv: "f753c5685daf86efa395504764e5a888"
server_key_version: "encryptionkey@0"
I20260620 23:11:21.681901 15724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:21.687352 15724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:21.688045 15724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:21.688251 15724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:21.689072 15724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:11:21.690418 15724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:11:21.690502 15724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:21.690647 15724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:11:21.690727 15724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:21.721324 15724 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.193:42173
I20260620 23:11:21.721441 15850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.193:42173 every 8 connection(s)
I20260620 23:11:21.722585 15724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
I20260620 23:11:21.730319 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 15724
I20260620 23:11:21.730501 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance
I20260620 23:11:21.730682 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:21.725266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:44765 (local address 127.10.191.254:44225)
0620 23:11:21.725676 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:21.725680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:21.726849 (+ 1169us) server_negotiation.cc:407] Connection header received
0620 23:11:21.728525 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:21.728529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:21.728580 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:21.728692 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:21.729893 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:21.729899 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:21.729904 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:11:21.729932 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:21.729988 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:11:21.730005 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:21.730158 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:21.730296 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 23:11:21.730360 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":216,"threads_started":1}
I20260620 23:11:21.731278 11007 external_mini_cluster.cc:1428] Setting key 76fbe62700bb5e3f55dabf7a61a8f797
I20260620 23:11:21.731628 15854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:21.725873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:44225 (local address 127.10.191.193:44765)
0620 23:11:21.726704 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:21.726744 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:21.728248 (+ 1504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:21.728879 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:21.728886 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:21.728973 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:21.729585 (+ 612us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:11:21.729587 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:21.729622 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:11:21.729701 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:21.730109 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:21.730115 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:21.730120 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:11:21.730317 (+ 197us) client_negotiation.cc:240] Negotiation successful
0620 23:11:21.730591 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":240,"threads_started":1}
I20260620 23:11:21.733220 15851 heartbeater.cc:347] Connected to a master server at 127.10.191.254:44225
I20260620 23:11:21.733553 15851 heartbeater.cc:464] Registering TS with master...
I20260620 23:11:21.734107 15851 heartbeater.cc:511] Master 127.10.191.254:44225 requested a full tablet report, sending...
I20260620 23:11:21.734270 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.194:0
--local_ip_for_outbound_sockets=127.10.191.194
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:44225
--builtin_ntp_servers=127.10.191.212:45265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 23:11:21.736311 15665 ts_manager.cc:194] Registered new tserver with Master: 51aaf66d2a6446fdb2c96b03d03c4c33 (127.10.191.193:42173)
W20260620 23:11:22.046041 15855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:22.046317 15855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:22.046499 15855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:22.053658 15855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:22.053753 15855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:22.053928 15855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.194
I20260620 23:11:22.062844 15855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:45265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:44225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.194
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:22.064141 15855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:22.065928 15855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:22.078678 15860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:22.078650 15861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:22.078728 15863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:22.079656 15855 server_base.cc:1034] running on GCE node
I20260620 23:11:22.080291 15855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:22.081192 15855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:22.082437 15855 hybrid_clock.cc:648] HybridClock initialized: now 1781997082082397 us; error 52 us; skew 500 ppm
I20260620 23:11:22.082801 15855 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:11:22.085089 15855 webserver.cc:460] Webserver started at http://127.10.191.194:35423/ using document root <none> and password file <none>
I20260620 23:11:22.085753 15855 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:22.085881 15855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:22.086216 15855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:11:22.088954 15855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-1/data/instance:
uuid: "ff4e01c36cd24b169b0f9c8a9ac510bd"
format_stamp: "Formatted at 2026-06-20 23:11:22 on dist-test-slave-jkbq"
server_key: "d289d1da5d8ed1cc7800e6827c8347ca"
server_key_iv: "370c1dfe66ddd8379e2a6ada66de42cd"
server_key_version: "encryptionkey@0"
I20260620 23:11:22.089708 15855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance:
uuid: "ff4e01c36cd24b169b0f9c8a9ac510bd"
format_stamp: "Formatted at 2026-06-20 23:11:22 on dist-test-slave-jkbq"
server_key: "d289d1da5d8ed1cc7800e6827c8347ca"
server_key_iv: "370c1dfe66ddd8379e2a6ada66de42cd"
server_key_version: "encryptionkey@0"
I20260620 23:11:22.094404 15855 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260620 23:11:22.097720 15869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:22.099188 15855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:11:22.099347 15855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-1/wal
uuid: "ff4e01c36cd24b169b0f9c8a9ac510bd"
format_stamp: "Formatted at 2026-06-20 23:11:22 on dist-test-slave-jkbq"
server_key: "d289d1da5d8ed1cc7800e6827c8347ca"
server_key_iv: "370c1dfe66ddd8379e2a6ada66de42cd"
server_key_version: "encryptionkey@0"
I20260620 23:11:22.099553 15855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:22.104984 15855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:22.105728 15855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:22.105985 15855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:22.106835 15855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:11:22.108181 15855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:11:22.108297 15855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:22.108430 15855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:11:22.108517 15855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:22.140867 15855 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.194:45869
I20260620 23:11:22.140923 15981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.194:45869 every 8 connection(s)
I20260620 23:11:22.142165 15855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
I20260620 23:11:22.150213 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:22.144702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:32963 (local address 127.10.191.254:44225)
0620 23:11:22.144879 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:22.144884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:22.146238 (+ 1354us) server_negotiation.cc:407] Connection header received
0620 23:11:22.147949 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:22.147954 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:22.148035 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:22.148157 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:22.149463 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:22.149469 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:22.149474 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:11:22.149494 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:22.149535 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:11:22.149547 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:22.149693 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:22.149867 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 23:11:22.149934 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:11:22.151204 15984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:22.145218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:44225 (local address 127.10.191.194:32963)
0620 23:11:22.146081 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:22.146121 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:22.147686 (+ 1565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:22.148382 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:22.148391 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:22.148482 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:22.149180 (+ 698us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:11:22.149182 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:22.149219 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:11:22.149300 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:22.149647 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:22.149660 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:22.149669 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 23:11:22.149949 (+ 280us) client_negotiation.cc:240] Negotiation successful
0620 23:11:22.150216 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":254,"threads_started":1}
I20260620 23:11:22.151829 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 15855
I20260620 23:11:22.151984 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance
I20260620 23:11:22.152499 11007 external_mini_cluster.cc:1428] Setting key f8a3fbf077a4fbe6522acca856a96de0
I20260620 23:11:22.152700 15982 heartbeater.cc:347] Connected to a master server at 127.10.191.254:44225
I20260620 23:11:22.153004 15982 heartbeater.cc:464] Registering TS with master...
I20260620 23:11:22.153697 15982 heartbeater.cc:511] Master 127.10.191.254:44225 requested a full tablet report, sending...
I20260620 23:11:22.155134 15665 ts_manager.cc:194] Registered new tserver with Master: ff4e01c36cd24b169b0f9c8a9ac510bd (127.10.191.194:45869)
I20260620 23:11:22.155272 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.195:0
--local_ip_for_outbound_sockets=127.10.191.195
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:44225
--builtin_ntp_servers=127.10.191.212:45265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 23:11:22.483408 15985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:22.483669 15985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:22.483757 15985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:22.490981 15985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:22.491070 15985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:22.491173 15985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.195
I20260620 23:11:22.499935 15985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:45265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:44225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.195
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:22.501091 15985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:22.502794 15985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:22.515450 15990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:22.515517 15993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:22.515542 15991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:22.516829 15985 server_base.cc:1034] running on GCE node
I20260620 23:11:22.517422 15985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:22.518338 15985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:22.519550 15985 hybrid_clock.cc:648] HybridClock initialized: now 1781997082519499 us; error 48 us; skew 500 ppm
I20260620 23:11:22.519876 15985 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:11:22.521997 15985 webserver.cc:460] Webserver started at http://127.10.191.195:34735/ using document root <none> and password file <none>
I20260620 23:11:22.522676 15985 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:22.522809 15985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:22.523142 15985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:11:22.525919 15985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-2/data/instance:
uuid: "b0c37c51b5d44122932a9b80e7d3eba9"
format_stamp: "Formatted at 2026-06-20 23:11:22 on dist-test-slave-jkbq"
server_key: "91a8a6491757ff23243fef86d99945fe"
server_key_iv: "eab105976ed1ef434d40c922f4abdfc7"
server_key_version: "encryptionkey@0"
I20260620 23:11:22.526677 15985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance:
uuid: "b0c37c51b5d44122932a9b80e7d3eba9"
format_stamp: "Formatted at 2026-06-20 23:11:22 on dist-test-slave-jkbq"
server_key: "91a8a6491757ff23243fef86d99945fe"
server_key_iv: "eab105976ed1ef434d40c922f4abdfc7"
server_key_version: "encryptionkey@0"
I20260620 23:11:22.531411 15985 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 23:11:22.534751 15999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:22.536036 15985 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:11:22.536207 15985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-2/wal
uuid: "b0c37c51b5d44122932a9b80e7d3eba9"
format_stamp: "Formatted at 2026-06-20 23:11:22 on dist-test-slave-jkbq"
server_key: "91a8a6491757ff23243fef86d99945fe"
server_key_iv: "eab105976ed1ef434d40c922f4abdfc7"
server_key_version: "encryptionkey@0"
I20260620 23:11:22.536433 15985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:22.541811 15985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:22.542527 15985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:22.542805 15985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:22.543673 15985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:11:22.545099 15985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:11:22.545213 15985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:22.545341 15985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:11:22.545428 15985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:22.576715 15985 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.195:37369
I20260620 23:11:22.576761 16111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.195:37369 every 8 connection(s)
I20260620 23:11:22.578016 15985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
I20260620 23:11:22.583369 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 15985
I20260620 23:11:22.583576 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance
I20260620 23:11:22.584110 11007 external_mini_cluster.cc:1428] Setting key bb828c633d7dd5090e15c5acf3b36fd4
I20260620 23:11:22.587605 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:22.580848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:46733 (local address 127.10.191.254:44225)
0620 23:11:22.581015 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:22.581020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:22.582785 (+ 1765us) server_negotiation.cc:407] Connection header received
0620 23:11:22.585107 (+ 2322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:22.585112 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:22.585194 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:22.585321 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:22.586830 (+ 1509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:22.586836 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:22.586840 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:11:22.586859 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:22.586918 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:11:22.586933 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:22.587056 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:22.587350 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 23:11:22.587409 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 23:11:22.588444 16114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:22.581607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:44225 (local address 127.10.191.195:46733)
0620 23:11:22.582633 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:22.582675 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:22.584854 (+ 2179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:22.585592 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:22.585600 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:22.585689 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:22.586469 (+ 780us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:11:22.586472 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:22.586542 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:11:22.586637 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:22.587107 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:22.587118 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:22.587125 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 23:11:22.587374 (+ 249us) client_negotiation.cc:240] Negotiation successful
0620 23:11:22.587559 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":351,"threads_started":1}
I20260620 23:11:22.589674 16112 heartbeater.cc:347] Connected to a master server at 127.10.191.254:44225
I20260620 23:11:22.589946 16112 heartbeater.cc:464] Registering TS with master...
I20260620 23:11:22.590479 16112 heartbeater.cc:511] Master 127.10.191.254:44225 requested a full tablet report, sending...
I20260620 23:11:22.591789 15665 ts_manager.cc:194] Registered new tserver with Master: b0c37c51b5d44122932a9b80e7d3eba9 (127.10.191.195:37369)
I20260620 23:11:22.599404 11007 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:11:22.606087 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:22.603795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53390 (local address 127.10.191.254:44225)
0620 23:11:22.603946 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:22.603950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:22.604546 (+ 596us) server_negotiation.cc:407] Connection header received
0620 23:11:22.604652 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:22.604655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:22.604702 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:22.604782 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:22.605564 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:22.605568 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:22.605571 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:11:22.605584 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:22.605612 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:11:22.605623 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:22.605713 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:22.605760 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 23:11:22.605801 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 23:11:22.609591 15665 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53390:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:11:22.616204 15665 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:11:22.634056 16124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:22.627312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.195:37369 (local address 127.0.0.1:52970)
0620 23:11:22.629112 (+ 1800us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:22.629172 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:22.629322 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:22.630799 (+ 1477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:22.630806 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:22.630915 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:22.631728 (+ 813us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:11:22.631731 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:22.631759 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:11:22.631832 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:22.633174 (+ 1342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:22.633186 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:22.633199 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 23:11:22.633387 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 23:11:22.633482 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1496,"thread_start_us":917,"threads_started":1}
I20260620 23:11:22.635358 16126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:22.627451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52970 (local address 127.10.191.195:37369)
0620 23:11:22.630233 (+ 2782us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:22.630241 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:22.630276 (+ 35us) server_negotiation.cc:407] Connection header received
0620 23:11:22.630351 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:22.630358 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:22.630597 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:22.631379 (+ 782us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:22.632932 (+ 1553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:22.632940 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:22.632953 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:11:22.632969 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:22.633027 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:11:22.633045 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:22.634799 (+ 1754us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:22.634888 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 23:11:22.634994 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2607,"thread_start_us":2447,"threads_started":1}
I20260620 23:11:22.635978 16127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:22.627956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35530 (local address 127.10.191.193:42173)
0620 23:11:22.629839 (+ 1883us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:22.629850 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:22.630492 (+ 642us) server_negotiation.cc:407] Connection header received
0620 23:11:22.630559 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:22.630565 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:22.632120 (+ 1555us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:22.632271 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:22.634067 (+ 1796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:22.634078 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:22.634092 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:11:22.634112 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:22.634168 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:11:22.634195 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:22.634321 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:22.635591 (+ 1270us) server_negotiation.cc:299] Negotiation successful
0620 23:11:22.635693 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1715,"thread_start_us":1612,"threads_started":1}
I20260620 23:11:22.636552 16125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:22.627432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:42173 (local address 127.0.0.1:35530)
0620 23:11:22.630271 (+ 2839us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:22.630293 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:22.630390 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:22.633051 (+ 2661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:22.633055 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:22.633080 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:22.633830 (+ 750us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:11:22.633833 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:22.633848 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:11:22.633895 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:22.634369 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:22.634373 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:22.634378 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:11:22.634526 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 23:11:22.634571 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2685,"thread_start_us":2554,"threads_started":1}
I20260620 23:11:22.640028 16124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:22.630367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.194:45869 (local address 127.0.0.1:37710)
0620 23:11:22.634282 (+ 3915us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:22.634299 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:22.634361 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:22.637724 (+ 3363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:22.637732 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:22.637782 (+ 50us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:22.638346 (+ 564us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:11:22.638352 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:22.638382 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:11:22.638439 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:22.639629 (+ 1190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:22.639634 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:22.639639 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:11:22.639753 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 23:11:22.639798 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3809,"thread_start_us":8993,"threads_started":1}
I20260620 23:11:22.639631 16047 tablet_service.cc:1467] Processing CreateTablet for tablet 5c6a085f651c41a68a6e2ca766dbf748 (DEFAULT_TABLE table=test-table [id=76ac6421cd024f50865d57ad57705a3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:11:22.640690 16129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:22.630738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37710 (local address 127.10.191.194:45869)
0620 23:11:22.637183 (+ 6445us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:22.637191 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:22.637219 (+ 28us) server_negotiation.cc:407] Connection header received
0620 23:11:22.637293 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:22.637303 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:22.637508 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:22.637661 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:22.638965 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:22.638972 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:22.638983 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:11:22.639004 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:22.639053 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:11:22.639071 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:22.639227 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:22.640232 (+ 1005us) server_negotiation.cc:299] Negotiation successful
0620 23:11:22.640327 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6282,"thread_start_us":6161,"threads_started":1}
I20260620 23:11:22.641220 16047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c6a085f651c41a68a6e2ca766dbf748. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:11:22.642665 15786 tablet_service.cc:1467] Processing CreateTablet for tablet 5c6a085f651c41a68a6e2ca766dbf748 (DEFAULT_TABLE table=test-table [id=76ac6421cd024f50865d57ad57705a3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:11:22.643939 15786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c6a085f651c41a68a6e2ca766dbf748. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:11:22.644392 15917 tablet_service.cc:1467] Processing CreateTablet for tablet 5c6a085f651c41a68a6e2ca766dbf748 (DEFAULT_TABLE table=test-table [id=76ac6421cd024f50865d57ad57705a3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:11:22.645687 15917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c6a085f651c41a68a6e2ca766dbf748. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:11:22.655134 16130 tablet_bootstrap.cc:492] T 5c6a085f651c41a68a6e2ca766dbf748 P 51aaf66d2a6446fdb2c96b03d03c4c33: Bootstrap starting.
I20260620 23:11:22.657114 16131 tablet_bootstrap.cc:492] T 5c6a085f651c41a68a6e2ca766dbf748 P b0c37c51b5d44122932a9b80e7d3eba9: Bootstrap starting.
I20260620 23:11:22.659993 16130 tablet_bootstrap.cc:654] T 5c6a085f651c41a68a6e2ca766dbf748 P 51aaf66d2a6446fdb2c96b03d03c4c33: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:22.660140 16131 tablet_bootstrap.cc:654] T 5c6a085f651c41a68a6e2ca766dbf748 P b0c37c51b5d44122932a9b80e7d3eba9: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:22.660715 16132 tablet_bootstrap.cc:492] T 5c6a085f651c41a68a6e2ca766dbf748 P ff4e01c36cd24b169b0f9c8a9ac510bd: Bootstrap starting.
I20260620 23:11:22.661216 16130 log.cc:824] T 5c6a085f651c41a68a6e2ca766dbf748 P 51aaf66d2a6446fdb2c96b03d03c4c33: Log is configured to *not* fsync() on all Append() calls
I20260620 23:11:22.661347 16131 log.cc:824] T 5c6a085f651c41a68a6e2ca766dbf748 P b0c37c51b5d44122932a9b80e7d3eba9: Log is configured to *not* fsync() on all Append() calls
I20260620 23:11:22.663919 16132 tablet_bootstrap.cc:654] T 5c6a085f651c41a68a6e2ca766dbf748 P ff4e01c36cd24b169b0f9c8a9ac510bd: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:22.664135 16131 tablet_bootstrap.cc:492] T 5c6a085f651c41a68a6e2ca766dbf748 P b0c37c51b5d44122932a9b80e7d3eba9: No bootstrap required, opened a new log
I20260620 23:11:22.664377 16131 ts_tablet_manager.cc:1397] T 5c6a085f651c41a68a6e2ca766dbf748 P b0c37c51b5d44122932a9b80e7d3eba9: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 23:11:22.664423 16130 tablet_bootstrap.cc:492] T 5c6a085f651c41a68a6e2ca766dbf748 P 51aaf66d2a6446fdb2c96b03d03c4c33: No bootstrap required, opened a new log
I20260620 23:11:22.664654 16130 ts_tablet_manager.cc:1397] T 5c6a085f651c41a68a6e2ca766dbf748 P 51aaf66d2a6446fdb2c96b03d03c4c33: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260620 23:11:22.665073 16132 log.cc:824] T 5c6a085f651c41a68a6e2ca766dbf748 P ff4e01c36cd24b169b0f9c8a9ac510bd: Log is configured to *not* fsync() on all Append() calls
I20260620 23:11:22.669075 16132 tablet_bootstrap.cc:492] T 5c6a085f651c41a68a6e2ca766dbf748 P ff4e01c36cd24b169b0f9c8a9ac510bd: No bootstrap required, opened a new log
I20260620 23:11:22.669314 16132 ts_tablet_manager.cc:1397] T 5c6a085f651c41a68a6e2ca766dbf748 P ff4e01c36cd24b169b0f9c8a9ac510bd: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 23:11:22.672057 16131 raft_consensus.cc:348] T 5c6a085f651c41a68a6e2ca766dbf748 P b0c37c51b5d44122932a9b80e7d3eba9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c37c51b5d44122932a9b80e7d3eba9" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 37369 } } peers { permanent_uuid: "51aaf66d2a6446fdb2c96b03d03c4c33" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 42173 } } peers { permanent_uuid: "ff4e01c36cd24b169b0f9c8a9ac510bd" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45869 } }
I20260620 23:11:22.672430 16131 raft_consensus.cc:374] T 5c6a085f651c41a68a6e2ca766dbf748 P b0c37c51b5d44122932a9b80e7d3eba9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:22.672518 16131 raft_consensus.cc:729] T 5c6a085f651c41a68a6e2ca766dbf748 P b0c37c51b5d44122932a9b80e7d3eba9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0c37c51b5d44122932a9b80e7d3eba9, State: Initialized, Role: FOLLOWER
I20260620 23:11:22.673095 16131 consensus_queue.cc:260] T 5c6a085f651c41a68a6e2ca766dbf748 P b0c37c51b5d44122932a9b80e7d3eba9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c37c51b5d44122932a9b80e7d3eba9" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 37369 } } peers { permanent_uuid: "51aaf66d2a6446fdb2c96b03d03c4c33" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 42173 } } peers { permanent_uuid: "ff4e01c36cd24b169b0f9c8a9ac510bd" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45869 } }
I20260620 23:11:22.675040 16130 raft_consensus.cc:348] T 5c6a085f651c41a68a6e2ca766dbf748 P 51aaf66d2a6446fdb2c96b03d03c4c33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c37c51b5d44122932a9b80e7d3eba9" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 37369 } } peers { permanent_uuid: "51aaf66d2a6446fdb2c96b03d03c4c33" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 42173 } } peers { permanent_uuid: "ff4e01c36cd24b169b0f9c8a9ac510bd" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45869 } }
I20260620 23:11:22.675446 16130 raft_consensus.cc:374] T 5c6a085f651c41a68a6e2ca766dbf748 P 51aaf66d2a6446fdb2c96b03d03c4c33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:22.675541 16130 raft_consensus.cc:729] T 5c6a085f651c41a68a6e2ca766dbf748 P 51aaf66d2a6446fdb2c96b03d03c4c33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51aaf66d2a6446fdb2c96b03d03c4c33, State: Initialized, Role: FOLLOWER
I20260620 23:11:22.676113 16130 consensus_queue.cc:260] T 5c6a085f651c41a68a6e2ca766dbf748 P 51aaf66d2a6446fdb2c96b03d03c4c33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c37c51b5d44122932a9b80e7d3eba9" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 37369 } } peers { permanent_uuid: "51aaf66d2a6446fdb2c96b03d03c4c33" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 42173 } } peers { permanent_uuid: "ff4e01c36cd24b169b0f9c8a9ac510bd" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45869 } }
I20260620 23:11:22.681064 16132 raft_consensus.cc:348] T 5c6a085f651c41a68a6e2ca766dbf748 P ff4e01c36cd24b169b0f9c8a9ac510bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c37c51b5d44122932a9b80e7d3eba9" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 37369 } } peers { permanent_uuid: "51aaf66d2a6446fdb2c96b03d03c4c33" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 42173 } } peers { permanent_uuid: "ff4e01c36cd24b169b0f9c8a9ac510bd" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45869 } }
I20260620 23:11:22.681607 16132 raft_consensus.cc:374] T 5c6a085f651c41a68a6e2ca766dbf748 P ff4e01c36cd24b169b0f9c8a9ac510bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:22.681721 16132 raft_consensus.cc:729] T 5c6a085f651c41a68a6e2ca766dbf748 P ff4e01c36cd24b169b0f9c8a9ac510bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff4e01c36cd24b169b0f9c8a9ac510bd, State: Initialized, Role: FOLLOWER
I20260620 23:11:22.682569 15851 heartbeater.cc:503] Master 127.10.191.254:44225 was elected leader, sending a full tablet report...
I20260620 23:11:22.682462 16132 consensus_queue.cc:260] T 5c6a085f651c41a68a6e2ca766dbf748 P ff4e01c36cd24b169b0f9c8a9ac510bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c37c51b5d44122932a9b80e7d3eba9" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 37369 } } peers { permanent_uuid: "51aaf66d2a6446fdb2c96b03d03c4c33" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 42173 } } peers { permanent_uuid: "ff4e01c36cd24b169b0f9c8a9ac510bd" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45869 } }
I20260620 23:11:22.683173 16130 ts_tablet_manager.cc:1428] T 5c6a085f651c41a68a6e2ca766dbf748 P 51aaf66d2a6446fdb2c96b03d03c4c33: Time spent starting tablet: real 0.018s user 0.012s sys 0.008s
I20260620 23:11:22.683974 16112 heartbeater.cc:503] Master 127.10.191.254:44225 was elected leader, sending a full tablet report...
I20260620 23:11:22.683967 15982 heartbeater.cc:503] Master 127.10.191.254:44225 was elected leader, sending a full tablet report...
I20260620 23:11:22.684584 16131 ts_tablet_manager.cc:1428] T 5c6a085f651c41a68a6e2ca766dbf748 P b0c37c51b5d44122932a9b80e7d3eba9: Time spent starting tablet: real 0.020s user 0.006s sys 0.015s
I20260620 23:11:22.685995 16132 ts_tablet_manager.cc:1428] T 5c6a085f651c41a68a6e2ca766dbf748 P ff4e01c36cd24b169b0f9c8a9ac510bd: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
W20260620 23:11:22.724848 15852 tablet.cc:2369] T 5c6a085f651c41a68a6e2ca766dbf748 P 51aaf66d2a6446fdb2c96b03d03c4c33: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:11:22.829988 16113 tablet.cc:2369] T 5c6a085f651c41a68a6e2ca766dbf748 P b0c37c51b5d44122932a9b80e7d3eba9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:11:22.889817 16136 raft_consensus.cc:482] T 5c6a085f651c41a68a6e2ca766dbf748 P b0c37c51b5d44122932a9b80e7d3eba9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:11:22.890110 16136 raft_consensus.cc:504] T 5c6a085f651c41a68a6e2ca766dbf748 P b0c37c51b5d44122932a9b80e7d3eba9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c37c51b5d44122932a9b80e7d3eba9" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 37369 } } peers { permanent_uuid: "51aaf66d2a6446fdb2c96b03d03c4c33" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 42173 } } peers { permanent_uuid: "ff4e01c36cd24b169b0f9c8a9ac510bd" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45869 } }
I20260620 23:11:22.891678 16136 leader_election.cc:288] T 5c6a085f651c41a68a6e2ca766dbf748 P b0c37c51b5d44122932a9b80e7d3eba9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51aaf66d2a6446fdb2c96b03d03c4c33 (127.10.191.193:42173), ff4e01c36cd24b169b0f9c8a9ac510bd (127.10.191.194:45869)
W20260620 23:11:22.894155 15983 tablet.cc:2369] T 5c6a085f651c41a68a6e2ca766dbf748 P ff4e01c36cd24b169b0f9c8a9ac510bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:11:22.894724 16114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:22.892002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:42173 (local address 127.10.191.195:58163)
0620 23:11:22.892176 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:22.892192 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:22.892313 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:22.892901 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:22.892906 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:22.892936 (+ 30us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:22.893604 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:11:22.893606 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:22.893619 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:11:22.893659 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:22.894302 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:22.894309 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:22.894315 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 23:11:22.894403 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 23:11:22.894455 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 23:11:22.894745 16127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:22.892227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:58163 (local address 127.10.191.193:42173)
0620 23:11:22.892438 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:22.892451 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:22.892476 (+ 25us) server_negotiation.cc:407] Connection header received
0620 23:11:22.892545 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:22.892551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:22.892613 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:22.892711 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:22.893807 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:22.893811 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:22.893814 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:11:22.893829 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:22.893857 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:11:22.893868 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:22.893985 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:22.894429 (+ 444us) server_negotiation.cc:299] Negotiation successful
0620 23:11:22.894483 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 23:11:22.895056 16129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:22.892684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:34821 (local address 127.10.191.194:45869)
0620 23:11:22.892867 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:22.892872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:22.893062 (+ 190us) server_negotiation.cc:407] Connection header received
0620 23:11:22.893149 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:22.893155 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:22.893221 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:22.893296 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:22.894312 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:22.894320 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:22.894327 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:11:22.894345 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:22.894376 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:11:22.894391 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:22.894507 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:22.894725 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 23:11:22.894782 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 23:11:22.895728 16139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:22.892361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.194:45869 (local address 127.10.191.195:34821)
0620 23:11:22.892955 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:22.892973 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:22.893061 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:22.893352 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:22.893358 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:22.893385 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:22.894126 (+ 741us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:11:22.894128 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:22.894140 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:11:22.894171 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:22.894563 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:22.894567 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:22.894571 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:11:22.895279 (+ 708us) client_negotiation.cc:240] Negotiation successful
0620 23:11:22.895329 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":333,"threads_started":1}
I20260620 23:11:22.895813 15806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c6a085f651c41a68a6e2ca766dbf748" candidate_uuid: "b0c37c51b5d44122932a9b80e7d3eba9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51aaf66d2a6446fdb2c96b03d03c4c33" is_pre_election: true
I20260620 23:11:22.896359 15806 raft_consensus.cc:2454] T 5c6a085f651c41a68a6e2ca766dbf748 P 51aaf66d2a6446fdb2c96b03d03c4c33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0c37c51b5d44122932a9b80e7d3eba9 in term 0.
I20260620 23:11:22.896762 15937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c6a085f651c41a68a6e2ca766dbf748" candidate_uuid: "b0c37c51b5d44122932a9b80e7d3eba9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff4e01c36cd24b169b0f9c8a9ac510bd" is_pre_election: true
I20260620 23:11:22.897024 16000 leader_election.cc:302] T 5c6a085f651c41a68a6e2ca766dbf748 P b0c37c51b5d44122932a9b80e7d3eba9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51aaf66d2a6446fdb2c96b03d03c4c33, b0c37c51b5d44122932a9b80e7d3eba9; no voters:
I20260620 23:11:22.897286 15937 raft_consensus.cc:2454] T 5c6a085f651c41a68a6e2ca766dbf748 P ff4e01c36cd24b169b0f9c8a9ac510bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0c37c51b5d44122932a9b80e7d3eba9 in term 0.
I20260620 23:11:22.897399 16136 raft_consensus.cc:2781] T 5c6a085f651c41a68a6e2ca766dbf748 P b0c37c51b5d44122932a9b80e7d3eba9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:11:22.897547 16136 raft_consensus.cc:482] T 5c6a085f651c41a68a6e2ca766dbf748 P b0c37c51b5d44122932a9b80e7d3eba9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:11:22.897635 16136 raft_consensus.cc:3037] T 5c6a085f651c41a68a6e2ca766dbf748 P b0c37c51b5d44122932a9b80e7d3eba9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:22.899858 16136 raft_consensus.cc:504] T 5c6a085f651c41a68a6e2ca766dbf748 P b0c37c51b5d44122932a9b80e7d3eba9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c37c51b5d44122932a9b80e7d3eba9" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 37369 } } peers { permanent_uuid: "51aaf66d2a6446fdb2c96b03d03c4c33" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 42173 } } peers { permanent_uuid: "ff4e01c36cd24b169b0f9c8a9ac510bd" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45869 } }
I20260620 23:11:22.900422 16136 leader_election.cc:288] T 5c6a085f651c41a68a6e2ca766dbf748 P b0c37c51b5d44122932a9b80e7d3eba9 [CANDIDATE]: Term 1 election: Requested vote from peers 51aaf66d2a6446fdb2c96b03d03c4c33 (127.10.191.193:42173), ff4e01c36cd24b169b0f9c8a9ac510bd (127.10.191.194:45869)
I20260620 23:11:22.900732 15806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c6a085f651c41a68a6e2ca766dbf748" candidate_uuid: "b0c37c51b5d44122932a9b80e7d3eba9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51aaf66d2a6446fdb2c96b03d03c4c33"
I20260620 23:11:22.900837 15937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c6a085f651c41a68a6e2ca766dbf748" candidate_uuid: "b0c37c51b5d44122932a9b80e7d3eba9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff4e01c36cd24b169b0f9c8a9ac510bd"
I20260620 23:11:22.900956 15806 raft_consensus.cc:3037] T 5c6a085f651c41a68a6e2ca766dbf748 P 51aaf66d2a6446fdb2c96b03d03c4c33 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:22.901028 15937 raft_consensus.cc:3037] T 5c6a085f651c41a68a6e2ca766dbf748 P ff4e01c36cd24b169b0f9c8a9ac510bd [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:22.903347 15806 raft_consensus.cc:2454] T 5c6a085f651c41a68a6e2ca766dbf748 P 51aaf66d2a6446fdb2c96b03d03c4c33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0c37c51b5d44122932a9b80e7d3eba9 in term 1.
I20260620 23:11:22.903558 15937 raft_consensus.cc:2454] T 5c6a085f651c41a68a6e2ca766dbf748 P ff4e01c36cd24b169b0f9c8a9ac510bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0c37c51b5d44122932a9b80e7d3eba9 in term 1.
I20260620 23:11:22.903774 16000 leader_election.cc:302] T 5c6a085f651c41a68a6e2ca766dbf748 P b0c37c51b5d44122932a9b80e7d3eba9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51aaf66d2a6446fdb2c96b03d03c4c33, b0c37c51b5d44122932a9b80e7d3eba9; no voters:
I20260620 23:11:22.904034 16136 raft_consensus.cc:2781] T 5c6a085f651c41a68a6e2ca766dbf748 P b0c37c51b5d44122932a9b80e7d3eba9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:11:22.904450 16136 raft_consensus.cc:686] T 5c6a085f651c41a68a6e2ca766dbf748 P b0c37c51b5d44122932a9b80e7d3eba9 [term 1 LEADER]: Becoming Leader. State: Replica: b0c37c51b5d44122932a9b80e7d3eba9, State: Running, Role: LEADER
I20260620 23:11:22.904868 16136 consensus_queue.cc:237] T 5c6a085f651c41a68a6e2ca766dbf748 P b0c37c51b5d44122932a9b80e7d3eba9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c37c51b5d44122932a9b80e7d3eba9" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 37369 } } peers { permanent_uuid: "51aaf66d2a6446fdb2c96b03d03c4c33" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 42173 } } peers { permanent_uuid: "ff4e01c36cd24b169b0f9c8a9ac510bd" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45869 } }
I20260620 23:11:22.909475 15665 catalog_manager.cc:5495] T 5c6a085f651c41a68a6e2ca766dbf748 P b0c37c51b5d44122932a9b80e7d3eba9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b0c37c51b5d44122932a9b80e7d3eba9 (127.10.191.195). New cstate: current_term: 1 leader_uuid: "b0c37c51b5d44122932a9b80e7d3eba9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c37c51b5d44122932a9b80e7d3eba9" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 37369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51aaf66d2a6446fdb2c96b03d03c4c33" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 42173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff4e01c36cd24b169b0f9c8a9ac510bd" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45869 } health_report { overall_health: UNKNOWN } } }
I20260620 23:11:22.927714 16126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:22.925830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52986 (local address 127.10.191.195:37369)
0620 23:11:22.925999 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:22.926004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:22.926020 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:11:22.926165 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:22.926169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:22.926241 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:22.926322 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:22.927155 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:22.927160 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:22.927163 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:11:22.927177 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:22.927206 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:11:22.927217 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:22.927317 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:22.927386 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 23:11:22.927427 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 23:11:22.935562 15806 raft_consensus.cc:1261] T 5c6a085f651c41a68a6e2ca766dbf748 P 51aaf66d2a6446fdb2c96b03d03c4c33 [term 1 FOLLOWER]: Refusing update from remote peer b0c37c51b5d44122932a9b80e7d3eba9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:11:22.935568 15937 raft_consensus.cc:1261] T 5c6a085f651c41a68a6e2ca766dbf748 P ff4e01c36cd24b169b0f9c8a9ac510bd [term 1 FOLLOWER]: Refusing update from remote peer b0c37c51b5d44122932a9b80e7d3eba9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:11:22.936455 16140 consensus_queue.cc:1035] T 5c6a085f651c41a68a6e2ca766dbf748 P b0c37c51b5d44122932a9b80e7d3eba9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51aaf66d2a6446fdb2c96b03d03c4c33" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 42173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:11:22.936817 16136 consensus_queue.cc:1035] T 5c6a085f651c41a68a6e2ca766dbf748 P b0c37c51b5d44122932a9b80e7d3eba9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff4e01c36cd24b169b0f9c8a9ac510bd" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 45869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:11:22.952708 16145 mvcc.cc:205] Tried to move back new op lower bound from 7299060051694198784 to 7299060051581566976. Current Snapshot: MvccSnapshot[applied={T|T < 7299060051694198784}]
I20260620 23:11:22.967446 15665 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:11:22.967718 15665 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:11:22.970913 15665 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5fa53d42498c4c3498190660e8b43f44: Sending DeleteTablet for 3 replicas of tablet 5c6a085f651c41a68a6e2ca766dbf748
I20260620 23:11:22.971935 16047 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c6a085f651c41a68a6e2ca766dbf748 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:11:22 UTC) from {username='slave'} at 127.0.0.1:52970
I20260620 23:11:22.972450 15786 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c6a085f651c41a68a6e2ca766dbf748 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:11:22 UTC) from {username='slave'} at 127.0.0.1:35530
I20260620 23:11:22.973099 15917 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c6a085f651c41a68a6e2ca766dbf748 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:11:22 UTC) from {username='slave'} at 127.0.0.1:37710
I20260620 23:11:22.973481 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 15724
I20260620 23:11:22.975991 16155 tablet_replica.cc:330] T 5c6a085f651c41a68a6e2ca766dbf748 P ff4e01c36cd24b169b0f9c8a9ac510bd: stopping tablet replica
I20260620 23:11:22.976459 16155 raft_consensus.cc:2229] T 5c6a085f651c41a68a6e2ca766dbf748 P ff4e01c36cd24b169b0f9c8a9ac510bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:11:22.976962 16155 raft_consensus.cc:2258] T 5c6a085f651c41a68a6e2ca766dbf748 P ff4e01c36cd24b169b0f9c8a9ac510bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:11:22.977732 16154 tablet_replica.cc:330] T 5c6a085f651c41a68a6e2ca766dbf748 P 51aaf66d2a6446fdb2c96b03d03c4c33: stopping tablet replica
I20260620 23:11:22.978567 16153 tablet_replica.cc:330] T 5c6a085f651c41a68a6e2ca766dbf748 P b0c37c51b5d44122932a9b80e7d3eba9: stopping tablet replica
I20260620 23:11:22.978619 16154 raft_consensus.cc:2229] T 5c6a085f651c41a68a6e2ca766dbf748 P 51aaf66d2a6446fdb2c96b03d03c4c33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:11:22.978957 16153 raft_consensus.cc:2229] T 5c6a085f651c41a68a6e2ca766dbf748 P b0c37c51b5d44122932a9b80e7d3eba9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:11:22.979417 16154 raft_consensus.cc:2258] T 5c6a085f651c41a68a6e2ca766dbf748 P 51aaf66d2a6446fdb2c96b03d03c4c33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:11:22.979601 16153 raft_consensus.cc:2258] T 5c6a085f651c41a68a6e2ca766dbf748 P b0c37c51b5d44122932a9b80e7d3eba9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:11:22.981279 16154 ts_tablet_manager.cc:1905] T 5c6a085f651c41a68a6e2ca766dbf748 P 51aaf66d2a6446fdb2c96b03d03c4c33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:11:22.981279 16155 ts_tablet_manager.cc:1905] T 5c6a085f651c41a68a6e2ca766dbf748 P ff4e01c36cd24b169b0f9c8a9ac510bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:11:22.981489 16153 ts_tablet_manager.cc:1905] T 5c6a085f651c41a68a6e2ca766dbf748 P b0c37c51b5d44122932a9b80e7d3eba9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:11:22.982513 16127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:22.974776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35540 (local address 127.10.191.193:42173)
0620 23:11:22.975156 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:22.975165 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:22.977922 (+ 2757us) server_negotiation.cc:407] Connection header received
0620 23:11:22.978129 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:22.978137 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:22.978259 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:22.978360 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:22.981653 (+ 3293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:22.981660 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:22.981665 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:11:22.981683 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:22.981721 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:11:22.981757 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:22.981874 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:22.982161 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 23:11:22.982219 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230}
I20260620 23:11:22.983723 15846 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:11:22.990170 16155 ts_tablet_manager.cc:1918] T 5c6a085f651c41a68a6e2ca766dbf748 P ff4e01c36cd24b169b0f9c8a9ac510bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:11:22.990360 16155 log.cc:1196] T 5c6a085f651c41a68a6e2ca766dbf748 P ff4e01c36cd24b169b0f9c8a9ac510bd: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-1/wal/wals/5c6a085f651c41a68a6e2ca766dbf748
I20260620 23:11:22.990856 16153 ts_tablet_manager.cc:1918] T 5c6a085f651c41a68a6e2ca766dbf748 P b0c37c51b5d44122932a9b80e7d3eba9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:11:22.991034 16153 log.cc:1196] T 5c6a085f651c41a68a6e2ca766dbf748 P b0c37c51b5d44122932a9b80e7d3eba9: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-2/wal/wals/5c6a085f651c41a68a6e2ca766dbf748
I20260620 23:11:22.991043 16155 ts_tablet_manager.cc:1939] T 5c6a085f651c41a68a6e2ca766dbf748 P ff4e01c36cd24b169b0f9c8a9ac510bd: Deleting consensus metadata
I20260620 23:11:22.991559 16153 ts_tablet_manager.cc:1939] T 5c6a085f651c41a68a6e2ca766dbf748 P b0c37c51b5d44122932a9b80e7d3eba9: Deleting consensus metadata
I20260620 23:11:22.993024 15649 catalog_manager.cc:4841] TS ff4e01c36cd24b169b0f9c8a9ac510bd (127.10.191.194:45869): tablet 5c6a085f651c41a68a6e2ca766dbf748 (table test-table [id=76ac6421cd024f50865d57ad57705a3d]) successfully deleted
I20260620 23:11:22.993513 15651 catalog_manager.cc:4841] TS b0c37c51b5d44122932a9b80e7d3eba9 (127.10.191.195:37369): tablet 5c6a085f651c41a68a6e2ca766dbf748 (table test-table [id=76ac6421cd024f50865d57ad57705a3d]) successfully deleted
I20260620 23:11:23.082212 16154 ts_tablet_manager.cc:1918] T 5c6a085f651c41a68a6e2ca766dbf748 P 51aaf66d2a6446fdb2c96b03d03c4c33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:11:23.082420 16154 log.cc:1196] T 5c6a085f651c41a68a6e2ca766dbf748 P 51aaf66d2a6446fdb2c96b03d03c4c33: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781997042893763-11007-0/minicluster-data/ts-0/wal/wals/5c6a085f651c41a68a6e2ca766dbf748
I20260620 23:11:23.083034 16154 ts_tablet_manager.cc:1939] T 5c6a085f651c41a68a6e2ca766dbf748 P 51aaf66d2a6446fdb2c96b03d03c4c33: Deleting consensus metadata
I20260620 23:11:23.083632 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 15724
I20260620 23:11:23.101075 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 15855
W20260620 23:11:23.101135 15649 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.191.193:42173 (error 108)
I20260620 23:11:23.103211 16124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:23.102561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:42173 (local address 127.0.0.1:35546)
0620 23:11:23.102854 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:23.103003 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.193:42173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182}
W20260620 23:11:23.103438 15649 catalog_manager.cc:4584] TS 51aaf66d2a6446fdb2c96b03d03c4c33 (127.10.191.193:42173): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5c6a085f651c41a68a6e2ca766dbf748: Network error: Client connection negotiation failed: client connection to 127.10.191.193:42173: connect: Connection refused (error 111)
I20260620 23:11:23.104703 16129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:23.102267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37712 (local address 127.10.191.194:45869)
0620 23:11:23.102437 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:23.102442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:23.102458 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:11:23.102511 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:23.102515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:23.102664 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:23.102789 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:23.104079 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:23.104085 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:23.104090 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:11:23.104107 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:23.104147 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:11:23.104158 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:23.104290 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:23.104384 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 23:11:23.104441 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:11:23.105340 15977 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:11:23.157152 16124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:23.156609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:42173 (local address 127.0.0.1:35552)
0620 23:11:23.156851 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:23.156965 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.193:42173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 23:11:23.158268 16124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:23.157876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:42173 (local address 127.0.0.1:35564)
0620 23:11:23.158041 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:23.158135 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.193:42173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 23:11:23.204551 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 15855
I20260620 23:11:23.219277 16124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:23.218619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:42173 (local address 127.0.0.1:35566)
0620 23:11:23.218941 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:23.219071 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.193:42173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260620 23:11:23.220675 16124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:23.220196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:42173 (local address 127.0.0.1:35568)
0620 23:11:23.220387 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:23.220492 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.193:42173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 23:11:23.223749 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 15985
I20260620 23:11:23.226843 16126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:23.224694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52998 (local address 127.10.191.195:37369)
0620 23:11:23.224894 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:23.224900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:23.224922 (+ 22us) server_negotiation.cc:407] Connection header received
0620 23:11:23.225007 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:23.225011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:23.225057 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:23.225172 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:23.226116 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:23.226126 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:23.226137 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:11:23.226155 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:23.226226 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:11:23.226255 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:23.226418 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:23.226484 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 23:11:23.226542 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 23:11:23.227627 16107 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:11:23.301259 16124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:23.300719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:42173 (local address 127.0.0.1:35572)
0620 23:11:23.300955 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:23.301049 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.193:42173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260620 23:11:23.302322 16124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:23.301950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:42173 (local address 127.0.0.1:35574)
0620 23:11:23.302124 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:23.302200 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.193:42173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260620 23:11:23.345690 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 15985
I20260620 23:11:23.364159 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 15634
I20260620 23:11:23.365339 15695 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:11:23.469262 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 15634
2026-06-20T23:11:23Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2725 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 23:11:23.509469 11007 mini_oidc.cc:152] Starting JWKS server
I20260620 23:11:23.512104 11007 webserver.cc:460] Webserver started at https://127.0.0.1:42515/ using document root <none> and password file <none>
I20260620 23:11:23.512423 11007 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 23:11:23.513053 11007 webserver.cc:460] Webserver started at http://127.0.0.1:43317/ using document root <none> and password file <none>
2026-06-20T23:11:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:11:23Z Disabled control of system clock
I20260620 23:11:23.537290 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:41657
--webserver_interface=127.10.191.254
--webserver_port=0
--builtin_ntp_servers=127.10.191.212:41227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:41657
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42515/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997042893763-11007-0/testchainca.cert with env {}
W20260620 23:11:23.847550 16167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:23.847880 16167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:23.848045 16167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:23.854954 16167 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:11:23.855093 16167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:23.855183 16167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:23.855273 16167 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:11:23.855357 16167 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 23:11:23.855445 16167 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 23:11:23.855525 16167 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42515/jwks.json
I20260620 23:11:23.865077 16167 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:41227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:41657
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:41657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42515/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997042893763-11007-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:23.866380 16167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:23.867698 16167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:23.879254 16172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:23.879168 16173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:23.879222 16175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:23.880040 16167 server_base.cc:1034] running on GCE node
I20260620 23:11:23.880801 16167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:23.882087 16167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:23.883316 16167 hybrid_clock.cc:648] HybridClock initialized: now 1781997083883283 us; error 47 us; skew 500 ppm
I20260620 23:11:23.883658 16167 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:11:23.886021 16167 webserver.cc:460] Webserver started at http://127.10.191.254:35287/ using document root <none> and password file <none>
I20260620 23:11:23.886665 16167 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:23.886796 16167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:23.887107 16167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:11:23.889763 16167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "3cc0a282900b48de84db094d26c5d558"
format_stamp: "Formatted at 2026-06-20 23:11:23 on dist-test-slave-jkbq"
server_key: "782bd820fe7e584b213191053ad611d1"
server_key_iv: "3fd372c280f5cef41bafcc80449be6bc"
server_key_version: "encryptionkey@0"
I20260620 23:11:23.890523 16167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "3cc0a282900b48de84db094d26c5d558"
format_stamp: "Formatted at 2026-06-20 23:11:23 on dist-test-slave-jkbq"
server_key: "782bd820fe7e584b213191053ad611d1"
server_key_iv: "3fd372c280f5cef41bafcc80449be6bc"
server_key_version: "encryptionkey@0"
I20260620 23:11:23.895243 16167 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:11:23.898392 16181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:23.899765 16167 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:11:23.899962 16167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "3cc0a282900b48de84db094d26c5d558"
format_stamp: "Formatted at 2026-06-20 23:11:23 on dist-test-slave-jkbq"
server_key: "782bd820fe7e584b213191053ad611d1"
server_key_iv: "3fd372c280f5cef41bafcc80449be6bc"
server_key_version: "encryptionkey@0"
I20260620 23:11:23.900187 16167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:23.920670 16167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:23.921649 16167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:23.921962 16167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:23.939633 16167 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:41657
I20260620 23:11:23.939651 16232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:41657 every 8 connection(s)
I20260620 23:11:23.940917 16167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:11:23.943011 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 16167
I20260620 23:11:23.943236 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:11:23.943820 11007 external_mini_cluster.cc:1428] Setting key 5201f20ad45472610b1bbb2f10fc3bfb
I20260620 23:11:23.946070 16233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:11:23.956185 16236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:23.945411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34848 (local address 127.10.191.254:41657)
0620 23:11:23.946303 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:23.946314 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:23.946360 (+ 46us) server_negotiation.cc:407] Connection header received
0620 23:11:23.947979 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:23.947999 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:23.948420 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:23.948834 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:23.949376 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:23.950702 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:23.952029 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:23.952633 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:23.952832 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:23.954014 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:23.954020 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:23.954032 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:11:23.954565 (+ 533us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:23.954635 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:11:23.954662 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:23.954784 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:23.955059 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 23:11:23.955332 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":294,"threads_started":1}
I20260620 23:11:23.956413 16233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cc0a282900b48de84db094d26c5d558: Bootstrap starting.
I20260620 23:11:23.959605 16233 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cc0a282900b48de84db094d26c5d558: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:23.959719 11007 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:11:23.960603 16233 log.cc:824] T 00000000000000000000000000000000 P 3cc0a282900b48de84db094d26c5d558: Log is configured to *not* fsync() on all Append() calls
I20260620 23:11:23.963375 16233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cc0a282900b48de84db094d26c5d558: No bootstrap required, opened a new log
I20260620 23:11:23.970314 16233 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cc0a282900b48de84db094d26c5d558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cc0a282900b48de84db094d26c5d558" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 41657 } }
I20260620 23:11:23.970659 16233 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3cc0a282900b48de84db094d26c5d558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:23.970765 16233 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cc0a282900b48de84db094d26c5d558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cc0a282900b48de84db094d26c5d558, State: Initialized, Role: FOLLOWER
I20260620 23:11:23.971303 16233 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cc0a282900b48de84db094d26c5d558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cc0a282900b48de84db094d26c5d558" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 41657 } }
I20260620 23:11:23.971539 16233 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3cc0a282900b48de84db094d26c5d558 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:11:23.971719 16233 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cc0a282900b48de84db094d26c5d558 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:11:23.971894 16233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cc0a282900b48de84db094d26c5d558 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:23.973889 16233 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cc0a282900b48de84db094d26c5d558 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cc0a282900b48de84db094d26c5d558" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 41657 } }
I20260620 23:11:23.974368 16233 leader_election.cc:302] T 00000000000000000000000000000000 P 3cc0a282900b48de84db094d26c5d558 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3cc0a282900b48de84db094d26c5d558; no voters:
I20260620 23:11:23.974993 16233 leader_election.cc:288] T 00000000000000000000000000000000 P 3cc0a282900b48de84db094d26c5d558 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:11:23.975129 16238 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cc0a282900b48de84db094d26c5d558 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:11:23.976326 16238 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3cc0a282900b48de84db094d26c5d558 [term 1 LEADER]: Becoming Leader. State: Replica: 3cc0a282900b48de84db094d26c5d558, State: Running, Role: LEADER
I20260620 23:11:23.976779 16238 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cc0a282900b48de84db094d26c5d558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cc0a282900b48de84db094d26c5d558" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 41657 } }
I20260620 23:11:23.977027 16233 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cc0a282900b48de84db094d26c5d558 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:11:23.979475 16240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cc0a282900b48de84db094d26c5d558 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cc0a282900b48de84db094d26c5d558. Latest consensus state: current_term: 1 leader_uuid: "3cc0a282900b48de84db094d26c5d558" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cc0a282900b48de84db094d26c5d558" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 41657 } } }
I20260620 23:11:23.979904 16240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cc0a282900b48de84db094d26c5d558 [sys.catalog]: This master's current role is: LEADER
I20260620 23:11:23.981911 16239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cc0a282900b48de84db094d26c5d558 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3cc0a282900b48de84db094d26c5d558" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cc0a282900b48de84db094d26c5d558" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 41657 } } }
I20260620 23:11:23.982510 16239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cc0a282900b48de84db094d26c5d558 [sys.catalog]: This master's current role is: LEADER
I20260620 23:11:23.983973 16245 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:11:23.989208 16245 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:11:23.997318 16245 catalog_manager.cc:1347] Generated new cluster ID: 069bf4d66b9942a2b749a0b844e2c839
I20260620 23:11:23.997428 16245 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:11:24.025418 16245 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:11:24.026686 16245 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:11:24.034289 16245 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3cc0a282900b48de84db094d26c5d558: Generated new TSK 0
I20260620 23:11:24.035182 16245 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 23:11:24.072904 16263 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 23:11:24.073084 16263 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 23:11:24.074255 16236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:24.052521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34858 (local address 127.10.191.254:41657)
0620 23:11:24.052776 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:24.052786 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:24.053390 (+ 604us) server_negotiation.cc:407] Connection header received
0620 23:11:24.053654 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:24.053661 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:24.053758 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:24.053916 (+ 158us) server_negotiation.cc:226] Negotiated authn=JWT
0620 23:11:24.054408 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:24.055562 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:24.056949 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:24.057664 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:24.057810 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:24.058277 (+ 467us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 23:11:24.073523 (+ 15246us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42515/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 23:11:24.073926 (+ 403us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34858: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42515/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 23:11:24.074383 16236 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34858: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42515/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 23:11:24.075624 16262 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:11:24.052435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41657 (local address 127.0.0.1:34858)
0620 23:11:24.053278 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:24.053300 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:24.053442 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:24.053956 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:24.053960 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:24.053970 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0620 23:11:24.054271 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:24.054279 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:24.055686 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:24.055690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:24.056811 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:24.056822 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:24.057845 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:24.057850 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:24.057960 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:24.058018 (+ 58us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 23:11:24.073835 (+ 15817us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42515/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 23:11:24.074103 (+ 268us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.191.254:41657: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42515/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":557,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.10.191.254:41657: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42515/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 23:11:24.078076 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 16167
I20260620 23:11:24.079290 16228 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:11:24.172547 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 16167
2026-06-20T23:11:24Z chronyd exiting
I20260620 23:11:24.223364 11007 test_util.cc:175] -----------------------------------------------
I20260620 23:11:24.223511 11007 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781997042893763-11007-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (718 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 23:11:24.227352 11007 mini_oidc.cc:152] Starting JWKS server
I20260620 23:11:24.229322 11007 webserver.cc:460] Webserver started at https://127.0.0.1:41583/ using document root <none> and password file <none>
I20260620 23:11:24.229562 11007 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 23:11:24.230219 11007 webserver.cc:460] Webserver started at http://127.0.0.1:41223/ using document root <none> and password file <none>
2026-06-20T23:11:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:11:24Z Disabled control of system clock
I20260620 23:11:24.256054 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:46335
--webserver_interface=127.10.191.254
--webserver_port=0
--builtin_ntp_servers=127.10.191.212:44719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:46335
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41583/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997042893763-11007-0/testchainca.cert with env {}
W20260620 23:11:24.571574 16270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:24.571921 16270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:24.572084 16270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:24.579173 16270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:11:24.579317 16270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:24.579407 16270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:24.579509 16270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:11:24.579592 16270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 23:11:24.579682 16270 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 23:11:24.579763 16270 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41583/jwks.json
I20260620 23:11:24.589819 16270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:44719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:46335
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:46335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41583/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997042893763-11007-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:24.591159 16270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:24.592489 16270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:24.603847 16276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:24.603883 16278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:24.604053 16275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:24.604794 16270 server_base.cc:1034] running on GCE node
I20260620 23:11:24.605574 16270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:24.606856 16270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:24.608114 16270 hybrid_clock.cc:648] HybridClock initialized: now 1781997084608077 us; error 34 us; skew 500 ppm
I20260620 23:11:24.608462 16270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:11:24.610767 16270 webserver.cc:460] Webserver started at http://127.10.191.254:34551/ using document root <none> and password file <none>
I20260620 23:11:24.611511 16270 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:24.611652 16270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:24.611984 16270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:11:24.614776 16270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "e24ecaf1a3694ddf9b474667c02e1e35"
format_stamp: "Formatted at 2026-06-20 23:11:24 on dist-test-slave-jkbq"
server_key: "730a514ece66213907b9cbf0c1fe862e"
server_key_iv: "f38d01902d0c1e0adee110a6aaa0e8fb"
server_key_version: "encryptionkey@0"
I20260620 23:11:24.615635 16270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "e24ecaf1a3694ddf9b474667c02e1e35"
format_stamp: "Formatted at 2026-06-20 23:11:24 on dist-test-slave-jkbq"
server_key: "730a514ece66213907b9cbf0c1fe862e"
server_key_iv: "f38d01902d0c1e0adee110a6aaa0e8fb"
server_key_version: "encryptionkey@0"
I20260620 23:11:24.620877 16270 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:11:24.624322 16284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:24.625710 16270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260620 23:11:24.625902 16270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "e24ecaf1a3694ddf9b474667c02e1e35"
format_stamp: "Formatted at 2026-06-20 23:11:24 on dist-test-slave-jkbq"
server_key: "730a514ece66213907b9cbf0c1fe862e"
server_key_iv: "f38d01902d0c1e0adee110a6aaa0e8fb"
server_key_version: "encryptionkey@0"
I20260620 23:11:24.626116 16270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:24.649240 16270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:24.650081 16270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:24.650400 16270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:24.668648 16270 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:46335
I20260620 23:11:24.668668 16335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:46335 every 8 connection(s)
I20260620 23:11:24.670037 16270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:11:24.672937 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 16270
I20260620 23:11:24.673122 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:11:24.673993 11007 external_mini_cluster.cc:1428] Setting key 59207b64e44c0b132d93e1daebd4ac04
I20260620 23:11:24.674335 16336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:11:24.687907 16339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:24.675878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51218 (local address 127.10.191.254:46335)
0620 23:11:24.676970 (+ 1092us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:24.676981 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:24.677046 (+ 65us) server_negotiation.cc:407] Connection header received
0620 23:11:24.679058 (+ 2012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:24.679079 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:24.679535 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:24.679863 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:24.680425 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:24.681733 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:24.683530 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:24.684183 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:24.684315 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:24.685517 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:24.685523 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:24.685540 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:11:24.686119 (+ 579us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:24.686194 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:11:24.686215 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:24.686312 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:24.686573 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 23:11:24.686983 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":388,"threads_started":1}
I20260620 23:11:24.688371 16336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e24ecaf1a3694ddf9b474667c02e1e35: Bootstrap starting.
I20260620 23:11:24.691756 16336 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e24ecaf1a3694ddf9b474667c02e1e35: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:24.691785 11007 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:11:24.692960 16336 log.cc:824] T 00000000000000000000000000000000 P e24ecaf1a3694ddf9b474667c02e1e35: Log is configured to *not* fsync() on all Append() calls
I20260620 23:11:24.695875 16336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e24ecaf1a3694ddf9b474667c02e1e35: No bootstrap required, opened a new log
I20260620 23:11:24.702991 16336 raft_consensus.cc:348] T 00000000000000000000000000000000 P e24ecaf1a3694ddf9b474667c02e1e35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e24ecaf1a3694ddf9b474667c02e1e35" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 46335 } }
I20260620 23:11:24.703331 16336 raft_consensus.cc:374] T 00000000000000000000000000000000 P e24ecaf1a3694ddf9b474667c02e1e35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:24.703402 16336 raft_consensus.cc:729] T 00000000000000000000000000000000 P e24ecaf1a3694ddf9b474667c02e1e35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e24ecaf1a3694ddf9b474667c02e1e35, State: Initialized, Role: FOLLOWER
I20260620 23:11:24.703900 16336 consensus_queue.cc:260] T 00000000000000000000000000000000 P e24ecaf1a3694ddf9b474667c02e1e35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e24ecaf1a3694ddf9b474667c02e1e35" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 46335 } }
I20260620 23:11:24.704087 16336 raft_consensus.cc:388] T 00000000000000000000000000000000 P e24ecaf1a3694ddf9b474667c02e1e35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:11:24.704239 16336 raft_consensus.cc:482] T 00000000000000000000000000000000 P e24ecaf1a3694ddf9b474667c02e1e35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:11:24.704404 16336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e24ecaf1a3694ddf9b474667c02e1e35 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:24.706775 16336 raft_consensus.cc:504] T 00000000000000000000000000000000 P e24ecaf1a3694ddf9b474667c02e1e35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e24ecaf1a3694ddf9b474667c02e1e35" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 46335 } }
I20260620 23:11:24.707360 16336 leader_election.cc:302] T 00000000000000000000000000000000 P e24ecaf1a3694ddf9b474667c02e1e35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e24ecaf1a3694ddf9b474667c02e1e35; no voters:
I20260620 23:11:24.707957 16336 leader_election.cc:288] T 00000000000000000000000000000000 P e24ecaf1a3694ddf9b474667c02e1e35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:11:24.708163 16341 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e24ecaf1a3694ddf9b474667c02e1e35 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:11:24.709028 16341 raft_consensus.cc:686] T 00000000000000000000000000000000 P e24ecaf1a3694ddf9b474667c02e1e35 [term 1 LEADER]: Becoming Leader. State: Replica: e24ecaf1a3694ddf9b474667c02e1e35, State: Running, Role: LEADER
I20260620 23:11:24.709575 16341 consensus_queue.cc:237] T 00000000000000000000000000000000 P e24ecaf1a3694ddf9b474667c02e1e35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e24ecaf1a3694ddf9b474667c02e1e35" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 46335 } }
I20260620 23:11:24.710107 16336 sys_catalog.cc:564] T 00000000000000000000000000000000 P e24ecaf1a3694ddf9b474667c02e1e35 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:11:24.712208 16343 sys_catalog.cc:455] T 00000000000000000000000000000000 P e24ecaf1a3694ddf9b474667c02e1e35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e24ecaf1a3694ddf9b474667c02e1e35. Latest consensus state: current_term: 1 leader_uuid: "e24ecaf1a3694ddf9b474667c02e1e35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e24ecaf1a3694ddf9b474667c02e1e35" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 46335 } } }
I20260620 23:11:24.712976 16343 sys_catalog.cc:458] T 00000000000000000000000000000000 P e24ecaf1a3694ddf9b474667c02e1e35 [sys.catalog]: This master's current role is: LEADER
I20260620 23:11:24.712495 16342 sys_catalog.cc:455] T 00000000000000000000000000000000 P e24ecaf1a3694ddf9b474667c02e1e35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e24ecaf1a3694ddf9b474667c02e1e35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e24ecaf1a3694ddf9b474667c02e1e35" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 46335 } } }
I20260620 23:11:24.713117 16342 sys_catalog.cc:458] T 00000000000000000000000000000000 P e24ecaf1a3694ddf9b474667c02e1e35 [sys.catalog]: This master's current role is: LEADER
I20260620 23:11:24.715397 16347 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:11:24.721087 16347 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:11:24.730134 16347 catalog_manager.cc:1347] Generated new cluster ID: 22dec825c8ba4d83acb15c18a9cdbff2
I20260620 23:11:24.730283 16347 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:11:24.741899 16347 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:11:24.743343 16347 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:11:24.750988 16347 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e24ecaf1a3694ddf9b474667c02e1e35: Generated new TSK 0
I20260620 23:11:24.751801 16347 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 23:11:24.790678 16366 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 23:11:24.790819 16366 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260620 23:11:24.792101 16365 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:11:24.772001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:46335 (local address 127.0.0.1:51222)
0620 23:11:24.772575 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:24.772590 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:24.772705 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:24.773175 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:24.773178 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:24.773187 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0620 23:11:24.773417 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:24.773424 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:24.774830 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:24.774833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:24.775967 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:24.775978 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:24.776973 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:24.776977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:24.777107 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:24.777138 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 23:11:24.791626 (+ 14488us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41583/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 23:11:24.791825 (+ 199us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.191.254:46335: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41583/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":318,"threads_started":1}
I20260620 23:11:24.792110 16339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:24.772140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51222 (local address 127.10.191.254:46335)
0620 23:11:24.772328 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:24.772335 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:24.772677 (+ 342us) server_negotiation.cc:407] Connection header received
0620 23:11:24.772886 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:24.772892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:24.772983 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:24.773164 (+ 181us) server_negotiation.cc:226] Negotiated authn=JWT
0620 23:11:24.773569 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:24.774704 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:24.776107 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:24.776816 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:24.776947 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:24.777410 (+ 463us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 23:11:24.791390 (+ 13980us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41583/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 23:11:24.791792 (+ 402us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51222: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41583/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":69}
W20260620 23:11:24.792241 16339 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51222: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41583/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260620 23:11:24.793546 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 16270
I20260620 23:11:24.794660 16331 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:11:24.885385 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 16270
2026-06-20T23:11:24Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (712 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 23:11:24.940053 11007 mini_oidc.cc:152] Starting JWKS server
I20260620 23:11:24.941854 11007 webserver.cc:460] Webserver started at https://127.0.0.1:37669/ using document root <none> and password file <none>
I20260620 23:11:24.942061 11007 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 23:11:24.942632 11007 webserver.cc:460] Webserver started at http://127.0.0.1:45297/ using document root <none> and password file <none>
2026-06-20T23:11:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:11:24Z Disabled control of system clock
I20260620 23:11:24.968803 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:34255
--webserver_interface=127.10.191.254
--webserver_port=0
--builtin_ntp_servers=127.10.191.212:40303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:34255
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37669/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 23:11:25.282687 16373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:25.283000 16373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:25.283090 16373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:25.290374 16373 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:11:25.290486 16373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:25.290524 16373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:25.290573 16373 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:11:25.290623 16373 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 23:11:25.290678 16373 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 23:11:25.290719 16373 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37669/jwks.json
I20260620 23:11:25.300246 16373 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:40303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:34255
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:34255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37669/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:25.301515 16373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:25.303367 16373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:25.315562 16378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:25.315585 16379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:25.315510 16381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:25.316635 16373 server_base.cc:1034] running on GCE node
I20260620 23:11:25.317448 16373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:25.319015 16373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:25.320248 16373 hybrid_clock.cc:648] HybridClock initialized: now 1781997085320202 us; error 41 us; skew 500 ppm
I20260620 23:11:25.320752 16373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:11:25.323220 16373 webserver.cc:460] Webserver started at http://127.10.191.254:37813/ using document root <none> and password file <none>
I20260620 23:11:25.323962 16373 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:25.324095 16373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:25.324424 16373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:11:25.327422 16373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "5d13cc80e29a4c959997b719e4b9f316"
format_stamp: "Formatted at 2026-06-20 23:11:25 on dist-test-slave-jkbq"
server_key: "d343faeecf759ec26a423d94a12d7ec0"
server_key_iv: "812855749f2eb3bdbac4997e5c7f6d4c"
server_key_version: "encryptionkey@0"
I20260620 23:11:25.328259 16373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "5d13cc80e29a4c959997b719e4b9f316"
format_stamp: "Formatted at 2026-06-20 23:11:25 on dist-test-slave-jkbq"
server_key: "d343faeecf759ec26a423d94a12d7ec0"
server_key_iv: "812855749f2eb3bdbac4997e5c7f6d4c"
server_key_version: "encryptionkey@0"
I20260620 23:11:25.333599 16373 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:11:25.337293 16387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:25.338693 16373 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 23:11:25.338965 16373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "5d13cc80e29a4c959997b719e4b9f316"
format_stamp: "Formatted at 2026-06-20 23:11:25 on dist-test-slave-jkbq"
server_key: "d343faeecf759ec26a423d94a12d7ec0"
server_key_iv: "812855749f2eb3bdbac4997e5c7f6d4c"
server_key_version: "encryptionkey@0"
I20260620 23:11:25.339197 16373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:25.361722 16373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:25.362648 16373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:25.363045 16373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:25.384021 16373 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:34255
I20260620 23:11:25.383888 16438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:34255 every 8 connection(s)
I20260620 23:11:25.385670 16373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:11:25.389628 16439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:11:25.395959 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 16373
I20260620 23:11:25.396203 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:11:25.397011 11007 external_mini_cluster.cc:1428] Setting key f969d0c4e55fb4e8406817be8b0754ea
I20260620 23:11:25.401525 16439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d13cc80e29a4c959997b719e4b9f316: Bootstrap starting.
I20260620 23:11:25.404775 16439 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d13cc80e29a4c959997b719e4b9f316: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:25.405786 16439 log.cc:824] T 00000000000000000000000000000000 P 5d13cc80e29a4c959997b719e4b9f316: Log is configured to *not* fsync() on all Append() calls
I20260620 23:11:25.408653 16439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d13cc80e29a4c959997b719e4b9f316: No bootstrap required, opened a new log
I20260620 23:11:25.410651 16442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:25.399139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44722 (local address 127.10.191.254:34255)
0620 23:11:25.400223 (+ 1084us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:25.400243 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:25.400286 (+ 43us) server_negotiation.cc:407] Connection header received
0620 23:11:25.402048 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:25.402072 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:25.402536 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:25.402989 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:25.403556 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:25.404796 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:25.406206 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:25.407009 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:25.407198 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:25.408362 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:25.408370 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:25.408384 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:11:25.408981 (+ 597us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:25.409067 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:11:25.409087 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:25.409221 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:25.409433 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 23:11:25.409728 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":588,"thread_start_us":361,"threads_started":1}
I20260620 23:11:25.414506 11007 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:11:25.417268 16439 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d13cc80e29a4c959997b719e4b9f316 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d13cc80e29a4c959997b719e4b9f316" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 34255 } }
I20260620 23:11:25.417690 16439 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d13cc80e29a4c959997b719e4b9f316 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:25.417798 16439 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d13cc80e29a4c959997b719e4b9f316 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d13cc80e29a4c959997b719e4b9f316, State: Initialized, Role: FOLLOWER
I20260620 23:11:25.418349 16439 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d13cc80e29a4c959997b719e4b9f316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d13cc80e29a4c959997b719e4b9f316" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 34255 } }
I20260620 23:11:25.418538 16439 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d13cc80e29a4c959997b719e4b9f316 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:11:25.418632 16439 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d13cc80e29a4c959997b719e4b9f316 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:11:25.418756 16439 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d13cc80e29a4c959997b719e4b9f316 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:25.421307 16439 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d13cc80e29a4c959997b719e4b9f316 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d13cc80e29a4c959997b719e4b9f316" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 34255 } }
I20260620 23:11:25.421730 16439 leader_election.cc:302] T 00000000000000000000000000000000 P 5d13cc80e29a4c959997b719e4b9f316 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d13cc80e29a4c959997b719e4b9f316; no voters:
I20260620 23:11:25.422245 16439 leader_election.cc:288] T 00000000000000000000000000000000 P 5d13cc80e29a4c959997b719e4b9f316 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:11:25.422400 16444 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d13cc80e29a4c959997b719e4b9f316 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:11:25.423174 16444 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d13cc80e29a4c959997b719e4b9f316 [term 1 LEADER]: Becoming Leader. State: Replica: 5d13cc80e29a4c959997b719e4b9f316, State: Running, Role: LEADER
I20260620 23:11:25.423707 16444 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d13cc80e29a4c959997b719e4b9f316 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d13cc80e29a4c959997b719e4b9f316" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 34255 } }
I20260620 23:11:25.424327 16439 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d13cc80e29a4c959997b719e4b9f316 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:11:25.426378 16446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d13cc80e29a4c959997b719e4b9f316 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d13cc80e29a4c959997b719e4b9f316. Latest consensus state: current_term: 1 leader_uuid: "5d13cc80e29a4c959997b719e4b9f316" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d13cc80e29a4c959997b719e4b9f316" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 34255 } } }
I20260620 23:11:25.426714 16446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d13cc80e29a4c959997b719e4b9f316 [sys.catalog]: This master's current role is: LEADER
I20260620 23:11:25.430631 16445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d13cc80e29a4c959997b719e4b9f316 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d13cc80e29a4c959997b719e4b9f316" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d13cc80e29a4c959997b719e4b9f316" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 34255 } } }
I20260620 23:11:25.431103 16445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d13cc80e29a4c959997b719e4b9f316 [sys.catalog]: This master's current role is: LEADER
I20260620 23:11:25.432714 16449 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:11:25.437067 16449 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:11:25.445466 16449 catalog_manager.cc:1347] Generated new cluster ID: d964128a3b504bcfa09a5d58295359ce
I20260620 23:11:25.445580 16449 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:11:25.462170 16449 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:11:25.463515 16449 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:11:25.473652 16449 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d13cc80e29a4c959997b719e4b9f316: Generated new TSK 0
I20260620 23:11:25.474453 16449 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:11:25.497735 16442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:25.489175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44730 (local address 127.10.191.254:34255)
0620 23:11:25.489370 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:25.489377 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:25.490144 (+ 767us) server_negotiation.cc:407] Connection header received
0620 23:11:25.490400 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:25.490406 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:25.490485 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:25.490640 (+ 155us) server_negotiation.cc:226] Negotiated authn=JWT
0620 23:11:25.491232 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:25.492015 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:25.493464 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:25.493987 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:25.494124 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:25.494637 (+ 513us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 23:11:25.497067 (+ 2430us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37669/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 23:11:25.497415 (+ 348us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44730: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37669/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 23:11:25.497855 16442 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44730: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37669/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 23:11:25.500087 16468 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:11:25.489117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:34255 (local address 127.0.0.1:44730)
0620 23:11:25.490035 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:25.490052 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:25.490228 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:25.490652 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:25.490658 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:25.490670 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0620 23:11:25.491066 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:25.491077 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:25.492162 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:25.492171 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:25.493313 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:25.493325 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:25.494207 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:25.494215 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:25.494358 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:25.494392 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 23:11:25.497294 (+ 2902us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37669/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 23:11:25.499620 (+ 2326us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.191.254:34255: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37669/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":788,"mutex_wait_us":2035,"thread_start_us":611,"threads_started":1}
I20260620 23:11:25.501919 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 16373
I20260620 23:11:25.503073 16434 generic_service.cc:196] Checking for leaks (request via RPC)
E20260620 23:11:25.510744 16469 webserver.cc:533] Webserver: sslize failed:
E20260620 23:11:25.510861 16469 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 23:11:25.592370 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 16373
2026-06-20T23:11:25Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (703 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T23:11:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:11:25Z Disabled control of system clock
I20260620 23:11:25.668936 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:36973
--webserver_interface=127.10.191.254
--webserver_port=0
--builtin_ntp_servers=127.10.191.212:36803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:36973
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997042893763-11007-0/testchainca.cert with env {}
W20260620 23:11:25.988122 16474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:25.988426 16474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:25.988516 16474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:25.995743 16474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:11:25.995848 16474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:25.995885 16474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:25.995934 16474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:11:25.995982 16474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 23:11:25.996037 16474 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 23:11:25.996078 16474 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 23:11:26.006009 16474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:36803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:36973
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:36973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997042893763-11007-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:26.007355 16474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:26.008630 16474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:26.020969 16480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:26.020942 16482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:26.021157 16479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:26.021854 16474 server_base.cc:1034] running on GCE node
I20260620 23:11:26.022651 16474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:26.024358 16474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:26.025614 16474 hybrid_clock.cc:648] HybridClock initialized: now 1781997086025565 us; error 55 us; skew 500 ppm
I20260620 23:11:26.026039 16474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:11:26.028638 16474 webserver.cc:460] Webserver started at http://127.10.191.254:43541/ using document root <none> and password file <none>
I20260620 23:11:26.029407 16474 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:26.029542 16474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:26.029871 16474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:11:26.032856 16474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "a60a149f7c7f42dab4dd91f05a57a891"
format_stamp: "Formatted at 2026-06-20 23:11:26 on dist-test-slave-jkbq"
server_key: "ec96c02280f3d94c5a058062f7280366"
server_key_iv: "82fdebf6dfdfdd350738f42f2dd8e831"
server_key_version: "encryptionkey@0"
I20260620 23:11:26.033725 16474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "a60a149f7c7f42dab4dd91f05a57a891"
format_stamp: "Formatted at 2026-06-20 23:11:26 on dist-test-slave-jkbq"
server_key: "ec96c02280f3d94c5a058062f7280366"
server_key_iv: "82fdebf6dfdfdd350738f42f2dd8e831"
server_key_version: "encryptionkey@0"
I20260620 23:11:26.038980 16474 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260620 23:11:26.042678 16488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:26.044149 16474 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 23:11:26.044375 16474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "a60a149f7c7f42dab4dd91f05a57a891"
format_stamp: "Formatted at 2026-06-20 23:11:26 on dist-test-slave-jkbq"
server_key: "ec96c02280f3d94c5a058062f7280366"
server_key_iv: "82fdebf6dfdfdd350738f42f2dd8e831"
server_key_version: "encryptionkey@0"
I20260620 23:11:26.044600 16474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:26.057600 16474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:26.058465 16474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:26.058756 16474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:26.077896 16474 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:36973
I20260620 23:11:26.077932 16539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:36973 every 8 connection(s)
I20260620 23:11:26.079274 16474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:11:26.083083 16540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:11:26.087158 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 16474
I20260620 23:11:26.087348 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:11:26.087983 11007 external_mini_cluster.cc:1428] Setting key c6bcea08aad9f366702faa48dd02294c
I20260620 23:11:26.095167 16540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a60a149f7c7f42dab4dd91f05a57a891: Bootstrap starting.
I20260620 23:11:26.098757 16540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a60a149f7c7f42dab4dd91f05a57a891: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:26.100034 16540 log.cc:824] T 00000000000000000000000000000000 P a60a149f7c7f42dab4dd91f05a57a891: Log is configured to *not* fsync() on all Append() calls
I20260620 23:11:26.103298 16540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a60a149f7c7f42dab4dd91f05a57a891: No bootstrap required, opened a new log
I20260620 23:11:26.103402 16543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:26.089889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46624 (local address 127.10.191.254:36973)
0620 23:11:26.090990 (+ 1101us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:26.091006 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:26.091060 (+ 54us) server_negotiation.cc:407] Connection header received
0620 23:11:26.093480 (+ 2420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:26.093509 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:26.094075 (+ 566us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:26.094517 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:26.095076 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:26.097021 (+ 1945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:26.098275 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:26.099388 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:26.099558 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:26.100734 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:26.100741 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:26.100758 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:11:26.101356 (+ 598us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:26.101450 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:11:26.101475 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:26.101584 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:26.101840 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 23:11:26.102147 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":312,"threads_started":1}
I20260620 23:11:26.107167 11007 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:11:26.111456 16540 raft_consensus.cc:348] T 00000000000000000000000000000000 P a60a149f7c7f42dab4dd91f05a57a891 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60a149f7c7f42dab4dd91f05a57a891" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 36973 } }
I20260620 23:11:26.111835 16540 raft_consensus.cc:374] T 00000000000000000000000000000000 P a60a149f7c7f42dab4dd91f05a57a891 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:26.111989 16540 raft_consensus.cc:729] T 00000000000000000000000000000000 P a60a149f7c7f42dab4dd91f05a57a891 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a60a149f7c7f42dab4dd91f05a57a891, State: Initialized, Role: FOLLOWER
I20260620 23:11:26.112583 16540 consensus_queue.cc:260] T 00000000000000000000000000000000 P a60a149f7c7f42dab4dd91f05a57a891 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60a149f7c7f42dab4dd91f05a57a891" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 36973 } }
I20260620 23:11:26.112826 16540 raft_consensus.cc:388] T 00000000000000000000000000000000 P a60a149f7c7f42dab4dd91f05a57a891 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:11:26.112962 16540 raft_consensus.cc:482] T 00000000000000000000000000000000 P a60a149f7c7f42dab4dd91f05a57a891 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:11:26.113145 16540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a60a149f7c7f42dab4dd91f05a57a891 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:26.115397 16540 raft_consensus.cc:504] T 00000000000000000000000000000000 P a60a149f7c7f42dab4dd91f05a57a891 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60a149f7c7f42dab4dd91f05a57a891" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 36973 } }
I20260620 23:11:26.115950 16540 leader_election.cc:302] T 00000000000000000000000000000000 P a60a149f7c7f42dab4dd91f05a57a891 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a60a149f7c7f42dab4dd91f05a57a891; no voters:
I20260620 23:11:26.116660 16540 leader_election.cc:288] T 00000000000000000000000000000000 P a60a149f7c7f42dab4dd91f05a57a891 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:11:26.116895 16545 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a60a149f7c7f42dab4dd91f05a57a891 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:11:26.117602 16545 raft_consensus.cc:686] T 00000000000000000000000000000000 P a60a149f7c7f42dab4dd91f05a57a891 [term 1 LEADER]: Becoming Leader. State: Replica: a60a149f7c7f42dab4dd91f05a57a891, State: Running, Role: LEADER
I20260620 23:11:26.118122 16545 consensus_queue.cc:237] T 00000000000000000000000000000000 P a60a149f7c7f42dab4dd91f05a57a891 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60a149f7c7f42dab4dd91f05a57a891" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 36973 } }
I20260620 23:11:26.118743 16540 sys_catalog.cc:564] T 00000000000000000000000000000000 P a60a149f7c7f42dab4dd91f05a57a891 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:11:26.120990 16546 sys_catalog.cc:455] T 00000000000000000000000000000000 P a60a149f7c7f42dab4dd91f05a57a891 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a60a149f7c7f42dab4dd91f05a57a891" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60a149f7c7f42dab4dd91f05a57a891" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 36973 } } }
I20260620 23:11:26.121424 16546 sys_catalog.cc:458] T 00000000000000000000000000000000 P a60a149f7c7f42dab4dd91f05a57a891 [sys.catalog]: This master's current role is: LEADER
I20260620 23:11:26.121985 16547 sys_catalog.cc:455] T 00000000000000000000000000000000 P a60a149f7c7f42dab4dd91f05a57a891 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a60a149f7c7f42dab4dd91f05a57a891. Latest consensus state: current_term: 1 leader_uuid: "a60a149f7c7f42dab4dd91f05a57a891" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60a149f7c7f42dab4dd91f05a57a891" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 36973 } } }
I20260620 23:11:26.122341 16547 sys_catalog.cc:458] T 00000000000000000000000000000000 P a60a149f7c7f42dab4dd91f05a57a891 [sys.catalog]: This master's current role is: LEADER
I20260620 23:11:26.124220 16553 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:11:26.129536 16553 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:11:26.137785 16553 catalog_manager.cc:1347] Generated new cluster ID: f69477be3abe45f2ab22739467502a27
I20260620 23:11:26.137903 16553 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:11:26.149453 16553 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:11:26.150755 16553 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:11:26.164697 16553 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a60a149f7c7f42dab4dd91f05a57a891: Generated new TSK 0
I20260620 23:11:26.165565 16553 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:11:26.320318 16543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:26.183180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46634 (local address 127.10.191.254:36973)
0620 23:11:26.183373 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:26.183380 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:26.183689 (+ 309us) server_negotiation.cc:407] Connection header received
0620 23:11:26.183917 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:26.183923 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:26.184012 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:26.184179 (+ 167us) server_negotiation.cc:226] Negotiated authn=JWT
0620 23:11:26.184613 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:26.185728 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:26.187237 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:26.187944 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:26.188066 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:26.188555 (+ 489us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 23:11:26.319545 (+130990us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 23:11:26.319967 (+ 422us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46634: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":80}
W20260620 23:11:26.320464 16543 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46634: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 23:11:26.320653 16569 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:11:26.183002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:36973 (local address 127.0.0.1:46634)
0620 23:11:26.183579 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:26.183597 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:26.183720 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:26.184210 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:26.184214 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:26.184223 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0620 23:11:26.184455 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:26.184463 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:26.185898 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:26.185902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:26.187095 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:26.187108 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:26.188127 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:26.188131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:26.188249 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:26.188283 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 23:11:26.319839 (+131556us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 23:11:26.320162 (+ 323us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.191.254:36973: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":313,"threads_started":1}
I20260620 23:11:26.322045 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 16474
I20260620 23:11:26.323215 16535 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:11:26.424798 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 16474
2026-06-20T23:11:26Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (816 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (395 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1464 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:11:28 dist-test-slave-jkbq krb5kdc[16585](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:11:28 dist-test-slave-jkbq krb5kdc[16585](info): set up 2 sockets
Jun 20 23:11:28 dist-test-slave-jkbq krb5kdc[16585](info): commencing operation
krb5kdc: starting...
W20260620 23:11:29.375468 11007 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Jun 20 23:11:29 dist-test-slave-jkbq krb5kdc[16585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:11:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:11:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.191.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
I20260620 23:11:29.590054 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:43383
--webserver_interface=127.10.191.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:40561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:43383
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:11:29.901497 16598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:29.901839 16598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:29.902009 16598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:29.909124 16598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:11:29.909274 16598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:29.909395 16598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:29.909502 16598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:11:29.909590 16598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:11:29.919135 16598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:40561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:43383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:43383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:29.920460 16598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:29.921851 16598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:29.933960 16603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:29.933915 16606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:29.934005 16604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:29.935150 16598 server_base.cc:1034] running on GCE node
I20260620 23:11:29.935894 16598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:29.937507 16598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:29.938751 16598 hybrid_clock.cc:648] HybridClock initialized: now 1781997089938671 us; error 63 us; skew 500 ppm
I20260620 23:11:29.939128 16598 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:11:29 dist-test-slave-jkbq krb5kdc[16585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997089, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:29.942293 16598 init.cc:377] Logged in from keytab as kudu/127.10.191.254@KRBTEST.COM (short username kudu)
I20260620 23:11:29.943848 16598 webserver.cc:460] Webserver started at http://127.10.191.254:33463/ using document root <none> and password file <none>
I20260620 23:11:29.944514 16598 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:29.944653 16598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:29.944978 16598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:11:29.947894 16598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "b3560e606b2a495a9f3242073314ba58"
format_stamp: "Formatted at 2026-06-20 23:11:29 on dist-test-slave-jkbq"
server_key: "794b63f471daae4ae3dd1955a10d1a7d"
server_key_iv: "fb8dbe94d1777e40834a01bc0edab4da"
server_key_version: "encryptionkey@0"
I20260620 23:11:29.948644 16598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "b3560e606b2a495a9f3242073314ba58"
format_stamp: "Formatted at 2026-06-20 23:11:29 on dist-test-slave-jkbq"
server_key: "794b63f471daae4ae3dd1955a10d1a7d"
server_key_iv: "fb8dbe94d1777e40834a01bc0edab4da"
server_key_version: "encryptionkey@0"
I20260620 23:11:29.953445 16598 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 23:11:29.956718 16613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:29.958012 16598 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260620 23:11:29.958187 16598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "b3560e606b2a495a9f3242073314ba58"
format_stamp: "Formatted at 2026-06-20 23:11:29 on dist-test-slave-jkbq"
server_key: "794b63f471daae4ae3dd1955a10d1a7d"
server_key_iv: "fb8dbe94d1777e40834a01bc0edab4da"
server_key_version: "encryptionkey@0"
I20260620 23:11:29.958374 16598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:29.972424 16598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:29.976651 16598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:29.976938 16598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:29.994798 16598 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:43383
I20260620 23:11:29.994819 16664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:43383 every 8 connection(s)
I20260620 23:11:29.996155 16598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:11:29.997997 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 16598
I20260620 23:11:29.998198 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:11:29.999049 11007 external_mini_cluster.cc:1428] Setting key 536149de5bf08460c9f7337f8b273057
I20260620 23:11:30.000706 16665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:11:30.011461 16665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3560e606b2a495a9f3242073314ba58: Bootstrap starting.
Jun 20 23:11:30 dist-test-slave-jkbq krb5kdc[16585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:11:30.014482 16665 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3560e606b2a495a9f3242073314ba58: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:30.015807 16665 log.cc:824] T 00000000000000000000000000000000 P b3560e606b2a495a9f3242073314ba58: Log is configured to *not* fsync() on all Append() calls
I20260620 23:11:30.018970 16665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3560e606b2a495a9f3242073314ba58: No bootstrap required, opened a new log
I20260620 23:11:30.020928 16668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:30.001062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33180 (local address 127.10.191.254:43383)
0620 23:11:30.001774 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:30.001786 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:30.001825 (+ 39us) server_negotiation.cc:407] Connection header received
0620 23:11:30.003493 (+ 1668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:30.003514 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:30.003894 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:30.004285 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:30.005659 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:30.006994 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:30.008182 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:30.008853 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:30.009063 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:30.012996 (+ 3933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:30.013007 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:30.013019 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:30.013062 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:30.016658 (+ 3596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:30.017391 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:30.017399 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:30.017406 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:30.017598 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:30.017935 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:30.017940 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:30.017943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:30.018568 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:30.018847 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:30.019236 (+ 389us) server_negotiation.cc:299] Negotiation successful
0620 23:11:30.019669 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":314,"threads_started":1}
I20260620 23:11:30.027211 16665 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3560e606b2a495a9f3242073314ba58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3560e606b2a495a9f3242073314ba58" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43383 } }
I20260620 23:11:30.027513 16665 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3560e606b2a495a9f3242073314ba58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:30.027582 16665 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3560e606b2a495a9f3242073314ba58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3560e606b2a495a9f3242073314ba58, State: Initialized, Role: FOLLOWER
I20260620 23:11:30.028028 16665 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3560e606b2a495a9f3242073314ba58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3560e606b2a495a9f3242073314ba58" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43383 } }
I20260620 23:11:30.028215 16665 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3560e606b2a495a9f3242073314ba58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:11:30.028306 16665 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3560e606b2a495a9f3242073314ba58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:11:30.028419 16665 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3560e606b2a495a9f3242073314ba58 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:30.030622 16665 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3560e606b2a495a9f3242073314ba58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3560e606b2a495a9f3242073314ba58" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43383 } }
I20260620 23:11:30.031088 16665 leader_election.cc:302] T 00000000000000000000000000000000 P b3560e606b2a495a9f3242073314ba58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3560e606b2a495a9f3242073314ba58; no voters:
I20260620 23:11:30.031589 16665 leader_election.cc:288] T 00000000000000000000000000000000 P b3560e606b2a495a9f3242073314ba58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:11:30.031723 16670 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3560e606b2a495a9f3242073314ba58 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:11:30.032375 16670 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3560e606b2a495a9f3242073314ba58 [term 1 LEADER]: Becoming Leader. State: Replica: b3560e606b2a495a9f3242073314ba58, State: Running, Role: LEADER
I20260620 23:11:30.032867 16670 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3560e606b2a495a9f3242073314ba58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3560e606b2a495a9f3242073314ba58" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43383 } }
I20260620 23:11:30.033476 16665 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3560e606b2a495a9f3242073314ba58 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:11:30.036130 16672 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3560e606b2a495a9f3242073314ba58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3560e606b2a495a9f3242073314ba58. Latest consensus state: current_term: 1 leader_uuid: "b3560e606b2a495a9f3242073314ba58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3560e606b2a495a9f3242073314ba58" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43383 } } }
I20260620 23:11:30.036559 16672 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3560e606b2a495a9f3242073314ba58 [sys.catalog]: This master's current role is: LEADER
I20260620 23:11:30.036821 16671 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3560e606b2a495a9f3242073314ba58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3560e606b2a495a9f3242073314ba58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3560e606b2a495a9f3242073314ba58" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43383 } } }
I20260620 23:11:30.040952 16671 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3560e606b2a495a9f3242073314ba58 [sys.catalog]: This master's current role is: LEADER
I20260620 23:11:30.048593 16679 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:11:30.052595 16679 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:11:30.060465 16679 catalog_manager.cc:1347] Generated new cluster ID: 97b46a6dfd414d6685105ade9b9e8576
I20260620 23:11:30.060561 16679 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:11:30.079493 16679 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:11:30.080701 16679 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:11:30.088747 16679 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3560e606b2a495a9f3242073314ba58: Generated new TSK 0
I20260620 23:11:30.089514 16679 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.191.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab.
Entry for principal kudu/127.10.191.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab.
I20260620 23:11:30.138368 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.193:0
--local_ip_for_outbound_sockets=127.10.191.193
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:43383
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:40561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:11:30.466003 16691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:30.466277 16691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:30.466436 16691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:30.473585 16691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:30.473691 16691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:30.473888 16691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.193
I20260620 23:11:30.483527 16691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:40561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:43383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.193
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:30.484827 16691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:30.486096 16691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:30.498670 16696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:30.498692 16697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:30.498747 16699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:30.499511 16691 server_base.cc:1034] running on GCE node
I20260620 23:11:30.500098 16691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:30.501003 16691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:30.502224 16691 hybrid_clock.cc:648] HybridClock initialized: now 1781997090502175 us; error 33 us; skew 500 ppm
I20260620 23:11:30.502566 16691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:11:30 dist-test-slave-jkbq krb5kdc[16585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997090, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:30.505658 16691 init.cc:377] Logged in from keytab as kudu/127.10.191.193@KRBTEST.COM (short username kudu)
I20260620 23:11:30.507160 16691 webserver.cc:460] Webserver started at http://127.10.191.193:40645/ using document root <none> and password file <none>
I20260620 23:11:30.507838 16691 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:30.507980 16691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:30.508302 16691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:11:30.511231 16691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-0/data/instance:
uuid: "6eebedfbb93047d5a555029687e93422"
format_stamp: "Formatted at 2026-06-20 23:11:30 on dist-test-slave-jkbq"
server_key: "2b941e71460d8cf8bc7e2714aa24aadb"
server_key_iv: "6dd00ea81a2b251003b1f1b861ab078a"
server_key_version: "encryptionkey@0"
I20260620 23:11:30.512004 16691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance:
uuid: "6eebedfbb93047d5a555029687e93422"
format_stamp: "Formatted at 2026-06-20 23:11:30 on dist-test-slave-jkbq"
server_key: "2b941e71460d8cf8bc7e2714aa24aadb"
server_key_iv: "6dd00ea81a2b251003b1f1b861ab078a"
server_key_version: "encryptionkey@0"
I20260620 23:11:30.517096 16691 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 23:11:30.520680 16706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:30.521993 16691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:11:30.522167 16691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-0/wal
uuid: "6eebedfbb93047d5a555029687e93422"
format_stamp: "Formatted at 2026-06-20 23:11:30 on dist-test-slave-jkbq"
server_key: "2b941e71460d8cf8bc7e2714aa24aadb"
server_key_iv: "6dd00ea81a2b251003b1f1b861ab078a"
server_key_version: "encryptionkey@0"
I20260620 23:11:30.522370 16691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:30.537910 16691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:30.542168 16691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:30.542459 16691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:30.543571 16691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:11:30.544870 16691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:11:30.544976 16691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:30.545128 16691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:11:30.545199 16691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:30.577313 16691 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.193:42171
I20260620 23:11:30.577358 16818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.193:42171 every 8 connection(s)
I20260620 23:11:30.578568 16691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
I20260620 23:11:30.587167 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 16691
I20260620 23:11:30.587360 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance
I20260620 23:11:30.588088 11007 external_mini_cluster.cc:1428] Setting key 01be345b6c27a6d296540d3e800e80f1
Jun 20 23:11:30 dist-test-slave-jkbq krb5kdc[16585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997090, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:11:30.597929 16821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:30.581168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:44715 (local address 127.10.191.254:43383)
0620 23:11:30.581613 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:30.581617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:30.582480 (+ 863us) server_negotiation.cc:407] Connection header received
0620 23:11:30.584337 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:30.584344 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:30.584434 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:30.584550 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:30.586001 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:30.586786 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:30.588468 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:30.589062 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:30.589200 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:30.592603 (+ 3403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:30.592609 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:30.592614 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:30.592654 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:30.595112 (+ 2458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:30.595789 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:30.595795 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:30.595800 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:30.595912 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:30.596310 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:30.596317 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:30.596321 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:30.596580 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:30.596730 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:30.597373 (+ 643us) server_negotiation.cc:299] Negotiation successful
0620 23:11:30.597581 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":252,"threads_started":1}
I20260620 23:11:30.598910 16822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:30.581617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43383 (local address 127.10.191.193:44715)
0620 23:11:30.582314 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:30.582358 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:30.584063 (+ 1705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:30.584798 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:30.584805 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:30.585294 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:30.585837 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:30.585852 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:30.586968 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:30.586972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:30.588328 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:30.588339 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:30.589275 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:30.589284 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:30.589517 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:30.590185 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:30.590206 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:30.592395 (+ 2189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:30.595243 (+ 2848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:30.595254 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:30.595268 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:30.595658 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:30.596018 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:30.596023 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:30.596027 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:30.596206 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:30.596704 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:30.596712 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:30.597032 (+ 320us) client_negotiation.cc:769] Sending connection context
0620 23:11:30.597290 (+ 258us) client_negotiation.cc:240] Negotiation successful
0620 23:11:30.597793 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":275,"threads_started":1}
I20260620 23:11:30.600457 16819 heartbeater.cc:347] Connected to a master server at 127.10.191.254:43383
I20260620 23:11:30.600775 16819 heartbeater.cc:464] Registering TS with master...
I20260620 23:11:30.601641 16819 heartbeater.cc:511] Master 127.10.191.254:43383 requested a full tablet report, sending...
I20260620 23:11:30.604130 16630 ts_manager.cc:194] Registered new tserver with Master: 6eebedfbb93047d5a555029687e93422 (127.10.191.193:42171)
I20260620 23:11:30.606257 16630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.193@KRBTEST.COM'} at 127.10.191.193:44715
WARNING: no policy specified for kudu/127.10.191.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab.
Entry for principal kudu/127.10.191.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab.
I20260620 23:11:30.632350 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.194:0
--local_ip_for_outbound_sockets=127.10.191.194
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:43383
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:40561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:11:30.945644 16825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:30.945909 16825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:30.946058 16825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:30.953378 16825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:30.953485 16825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:30.953698 16825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.194
I20260620 23:11:30.963800 16825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:40561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:43383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.194
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:30.965289 16825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:30.966647 16825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:30.979589 16831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:30.979563 16833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:30.979781 16830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:30.980366 16825 server_base.cc:1034] running on GCE node
I20260620 23:11:30.980937 16825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:30.981830 16825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:30.983044 16825 hybrid_clock.cc:648] HybridClock initialized: now 1781997090982999 us; error 68 us; skew 500 ppm
I20260620 23:11:30.983352 16825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:11:30 dist-test-slave-jkbq krb5kdc[16585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997090, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:30.986380 16825 init.cc:377] Logged in from keytab as kudu/127.10.191.194@KRBTEST.COM (short username kudu)
I20260620 23:11:30.987769 16825 webserver.cc:460] Webserver started at http://127.10.191.194:41769/ using document root <none> and password file <none>
I20260620 23:11:30.988448 16825 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:30.988590 16825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:30.988894 16825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:11:30.991739 16825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-1/data/instance:
uuid: "4311717cdbc744679d9abaf2b958dc9b"
format_stamp: "Formatted at 2026-06-20 23:11:30 on dist-test-slave-jkbq"
server_key: "a58937ee1615d27deaddb07987a73f3b"
server_key_iv: "84e2d9742fc80947809bf4e67ace8b56"
server_key_version: "encryptionkey@0"
I20260620 23:11:30.992539 16825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance:
uuid: "4311717cdbc744679d9abaf2b958dc9b"
format_stamp: "Formatted at 2026-06-20 23:11:30 on dist-test-slave-jkbq"
server_key: "a58937ee1615d27deaddb07987a73f3b"
server_key_iv: "84e2d9742fc80947809bf4e67ace8b56"
server_key_version: "encryptionkey@0"
I20260620 23:11:30.997601 16825 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:11:31.001076 16840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:31.002388 16825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:11:31.002552 16825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-1/wal
uuid: "4311717cdbc744679d9abaf2b958dc9b"
format_stamp: "Formatted at 2026-06-20 23:11:30 on dist-test-slave-jkbq"
server_key: "a58937ee1615d27deaddb07987a73f3b"
server_key_iv: "84e2d9742fc80947809bf4e67ace8b56"
server_key_version: "encryptionkey@0"
I20260620 23:11:31.002761 16825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:31.019618 16825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:31.023906 16825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:31.024194 16825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:31.025098 16825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:11:31.026486 16825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:11:31.026587 16825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:31.026736 16825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:11:31.026788 16825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:31.059605 16825 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.194:40433
I20260620 23:11:31.059643 16952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.194:40433 every 8 connection(s)
I20260620 23:11:31.060940 16825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
I20260620 23:11:31.070804 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 16825
I20260620 23:11:31.071020 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance
I20260620 23:11:31.071467 11007 external_mini_cluster.cc:1428] Setting key 8fa31dc43c3ff857c0f79a53ad8d1511
Jun 20 23:11:31 dist-test-slave-jkbq krb5kdc[16585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997090, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:11:31.081664 16821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:31.063898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:41447 (local address 127.10.191.254:43383)
0620 23:11:31.064071 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:31.064075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:31.065222 (+ 1147us) server_negotiation.cc:407] Connection header received
0620 23:11:31.067002 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:31.067007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:31.067111 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:31.067223 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:31.068668 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:31.069394 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:31.070927 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:31.071423 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:31.071561 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:31.075488 (+ 3927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:31.075493 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:31.075496 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:31.075527 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:31.078138 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:31.079077 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:31.079084 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:31.079089 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:31.079254 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:31.079685 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:31.079691 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:31.079695 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:31.080101 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:31.080249 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:31.080952 (+ 703us) server_negotiation.cc:299] Negotiation successful
0620 23:11:31.081209 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:11:31.082331 16955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:31.064355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43383 (local address 127.10.191.194:41447)
0620 23:11:31.065052 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:31.065095 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:31.066696 (+ 1601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:31.067458 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:31.067467 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:31.067962 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:31.068500 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:31.068516 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:31.069553 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:31.069557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:31.070766 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:31.070778 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:31.071650 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:31.071657 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:31.071887 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:31.072563 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:31.072589 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:31.075275 (+ 2686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:31.078286 (+ 3011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:31.078299 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:31.078319 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:31.078931 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:31.079392 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:31.079395 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:31.079397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:31.079566 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:31.080239 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:31.080248 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:31.080583 (+ 335us) client_negotiation.cc:769] Sending connection context
0620 23:11:31.080817 (+ 234us) client_negotiation.cc:240] Negotiation successful
0620 23:11:31.081200 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":291,"threads_started":1}
I20260620 23:11:31.083913 16953 heartbeater.cc:347] Connected to a master server at 127.10.191.254:43383
I20260620 23:11:31.084230 16953 heartbeater.cc:464] Registering TS with master...
I20260620 23:11:31.085111 16953 heartbeater.cc:511] Master 127.10.191.254:43383 requested a full tablet report, sending...
I20260620 23:11:31.086791 16630 ts_manager.cc:194] Registered new tserver with Master: 4311717cdbc744679d9abaf2b958dc9b (127.10.191.194:40433)
I20260620 23:11:31.087687 16630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.194@KRBTEST.COM'} at 127.10.191.194:41447
WARNING: no policy specified for kudu/127.10.191.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab.
Entry for principal kudu/127.10.191.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab.
I20260620 23:11:31.118170 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.195:0
--local_ip_for_outbound_sockets=127.10.191.195
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:43383
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:40561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:11:31.437978 16958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:31.438261 16958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:31.438366 16958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:31.445639 16958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:31.445742 16958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:31.445878 16958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.195
I20260620 23:11:31.456717 16958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:40561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:43383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.195
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:31.457989 16958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:31.459321 16958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:31.472349 16963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:31.472384 16964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:31.472433 16966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:31.473095 16958 server_base.cc:1034] running on GCE node
I20260620 23:11:31.473717 16958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:31.474617 16958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:31.475831 16958 hybrid_clock.cc:648] HybridClock initialized: now 1781997091475763 us; error 64 us; skew 500 ppm
I20260620 23:11:31.476150 16958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:11:31 dist-test-slave-jkbq krb5kdc[16585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997091, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:31.479290 16958 init.cc:377] Logged in from keytab as kudu/127.10.191.195@KRBTEST.COM (short username kudu)
I20260620 23:11:31.480701 16958 webserver.cc:460] Webserver started at http://127.10.191.195:38539/ using document root <none> and password file <none>
I20260620 23:11:31.481369 16958 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:31.481510 16958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:31.481804 16958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:11:31.484645 16958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-2/data/instance:
uuid: "908dc02db5fe4cd3a3669db2a35dc5ca"
format_stamp: "Formatted at 2026-06-20 23:11:31 on dist-test-slave-jkbq"
server_key: "cc4df0e68fd167210142e2aff28c88a0"
server_key_iv: "7ec903615819c309b118a3fbbb82d6b4"
server_key_version: "encryptionkey@0"
I20260620 23:11:31.485430 16958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance:
uuid: "908dc02db5fe4cd3a3669db2a35dc5ca"
format_stamp: "Formatted at 2026-06-20 23:11:31 on dist-test-slave-jkbq"
server_key: "cc4df0e68fd167210142e2aff28c88a0"
server_key_iv: "7ec903615819c309b118a3fbbb82d6b4"
server_key_version: "encryptionkey@0"
I20260620 23:11:31.490357 16958 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:11:31.494318 16973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:31.495784 16958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:11:31.495945 16958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-2/wal
uuid: "908dc02db5fe4cd3a3669db2a35dc5ca"
format_stamp: "Formatted at 2026-06-20 23:11:31 on dist-test-slave-jkbq"
server_key: "cc4df0e68fd167210142e2aff28c88a0"
server_key_iv: "7ec903615819c309b118a3fbbb82d6b4"
server_key_version: "encryptionkey@0"
I20260620 23:11:31.496150 16958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:31.520028 16958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:31.524246 16958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:31.524519 16958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:31.525456 16958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:11:31.526755 16958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:11:31.526825 16958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:31.526996 16958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:11:31.527066 16958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:31.560035 16958 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.195:38991
I20260620 23:11:31.560123 17085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.195:38991 every 8 connection(s)
I20260620 23:11:31.561287 16958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
I20260620 23:11:31.566781 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 16958
I20260620 23:11:31.566974 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance
I20260620 23:11:31.567425 11007 external_mini_cluster.cc:1428] Setting key e667dacca5fb4d0b2b68c885d8a6a28a
Jun 20 23:11:31 dist-test-slave-jkbq krb5kdc[16585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997091, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:11:31.581490 16821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:31.564058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:47217 (local address 127.10.191.254:43383)
0620 23:11:31.564234 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:31.564238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:31.565426 (+ 1188us) server_negotiation.cc:407] Connection header received
0620 23:11:31.567235 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:31.567240 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:31.567324 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:31.567447 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:31.569170 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:31.570078 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:31.571673 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:31.572051 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:31.572145 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:31.575383 (+ 3238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:31.575388 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:31.575391 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:31.575420 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:31.578444 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:31.579215 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:31.579220 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:31.579223 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:31.579398 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:31.579789 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:31.579794 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:31.579797 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:31.580114 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:31.580268 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:31.580944 (+ 676us) server_negotiation.cc:299] Negotiation successful
0620 23:11:31.581150 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:11:31.582659 17088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:31.564578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43383 (local address 127.10.191.195:47217)
0620 23:11:31.565266 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:31.565302 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:31.566910 (+ 1608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:31.567686 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:31.567694 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:31.568349 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:31.568989 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:31.569007 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:31.570241 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:31.570246 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:31.571546 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:31.571562 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:31.572234 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:31.572238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:31.572420 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:31.573084 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:31.573105 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:31.575207 (+ 2102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:31.578582 (+ 3375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:31.578589 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:31.578600 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:31.579091 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:31.579505 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:31.579509 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:31.579511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:31.579670 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:31.580245 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:31.580250 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:31.580522 (+ 272us) client_negotiation.cc:769] Sending connection context
0620 23:11:31.580831 (+ 309us) client_negotiation.cc:240] Negotiation successful
0620 23:11:31.581331 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":269,"threads_started":1}
I20260620 23:11:31.584205 17086 heartbeater.cc:347] Connected to a master server at 127.10.191.254:43383
I20260620 23:11:31.584460 17086 heartbeater.cc:464] Registering TS with master...
I20260620 23:11:31.585067 17086 heartbeater.cc:511] Master 127.10.191.254:43383 requested a full tablet report, sending...
I20260620 23:11:31.586297 16630 ts_manager.cc:194] Registered new tserver with Master: 908dc02db5fe4cd3a3669db2a35dc5ca (127.10.191.195:38991)
I20260620 23:11:31.587163 16630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.195@KRBTEST.COM'} at 127.10.191.195:47217
I20260620 23:11:31.594843 11007 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:11:31.609280 16819 heartbeater.cc:503] Master 127.10.191.254:43383 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/krb5kdc/test-admin.keytab.
Jun 20 23:11:31 dist-test-slave-jkbq krb5kdc[16585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:31.620762 11007 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 23:11:31.625648 17093 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:11:31.623174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50318 (local address 127.10.191.193:42171)
0620 23:11:31.623844 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:31.623855 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:31.623887 (+ 32us) server_negotiation.cc:407] Connection header received
0620 23:11:31.623956 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:31.623962 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:31.624166 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:31.624328 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:31.625193 (+ 865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50318: BlockingRecv error: recv got EOF from 127.0.0.1:50318 (error 108)
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":317,"threads_started":1}
W20260620 23:11:31.625692 11007 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/krb5kdc/krb5cc' of type FILE
Jun 20 23:11:31 dist-test-slave-jkbq krb5kdc[16585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:31.627492 11007 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 23:11:31 dist-test-slave-jkbq krb5kdc[16585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:31.628887 11007 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 23:11:31 dist-test-slave-jkbq krb5kdc[16585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.193@KRBTEST.COM
I20260620 23:11:31.643754 17093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:31.630472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50320 (local address 127.10.191.193:42171)
0620 23:11:31.630652 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:31.630658 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:31.630818 (+ 160us) server_negotiation.cc:407] Connection header received
0620 23:11:31.631037 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:31.631041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:31.631095 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:31.631178 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:31.632127 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:31.633081 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:31.634196 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:31.634819 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:31.634989 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:31.638509 (+ 3520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:31.638519 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:31.638534 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:31.638563 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:31.641357 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:31.641972 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:31.641979 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:31.641982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:31.642092 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:31.642407 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:31.642410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:31.642412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:31.642750 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:31.643051 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:31.643329 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 23:11:31.643511 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 23:11:31.649466 17093 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:11:31.647709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50326 (local address 127.10.191.193:42171)
0620 23:11:31.647872 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:31.647875 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:31.648280 (+ 405us) server_negotiation.cc:407] Connection header received
0620 23:11:31.648413 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:31.648417 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:31.648478 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:31.648548 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:31.649226 (+ 678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50326: BlockingRecv error: recv got EOF from 127.0.0.1:50326 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 23:11:31.649742 11007 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/krb5kdc/krb5cc' of type FILE
Jun 20 23:11:31 dist-test-slave-jkbq krb5kdc[16585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:31.651249 11007 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 23:11:31 dist-test-slave-jkbq krb5kdc[16585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:31.652498 11007 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 23:11:31 dist-test-slave-jkbq krb5kdc[16585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.193@KRBTEST.COM
I20260620 23:11:31.665987 17093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:31.653661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50330 (local address 127.10.191.193:42171)
0620 23:11:31.653849 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:31.653857 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:31.654131 (+ 274us) server_negotiation.cc:407] Connection header received
0620 23:11:31.654307 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:31.654316 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:31.654366 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:31.654462 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:31.655408 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:31.656097 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:31.657291 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:31.657726 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:31.657871 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:31.661103 (+ 3232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:31.661108 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:31.661111 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:31.661150 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:31.663637 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:31.664234 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:31.664244 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:31.664247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:31.664357 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:31.664727 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:31.664733 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:31.664736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:31.664991 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:31.665251 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:31.665526 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 23:11:31.665714 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 23:11:31.669929 17093 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:11:31.668384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50336 (local address 127.10.191.193:42171)
0620 23:11:31.668531 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:31.668535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:31.668854 (+ 319us) server_negotiation.cc:407] Connection header received
0620 23:11:31.668962 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:31.668966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:31.669026 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:31.669099 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:31.669660 (+ 561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50336: BlockingRecv error: recv got EOF from 127.0.0.1:50336 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260620 23:11:31.670095 11007 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781997042893763-11007-0/krb5kdc/krb5cc' of type FILE
Jun 20 23:11:31 dist-test-slave-jkbq krb5kdc[16585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:31.671586 11007 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 23:11:31 dist-test-slave-jkbq krb5kdc[16585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:31.672777 11007 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 23:11:31 dist-test-slave-jkbq krb5kdc[16585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.193@KRBTEST.COM
I20260620 23:11:31.685593 17093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:31.673929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50338 (local address 127.10.191.193:42171)
0620 23:11:31.674139 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:31.674145 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:31.674364 (+ 219us) server_negotiation.cc:407] Connection header received
0620 23:11:31.674473 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:31.674476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:31.674526 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:31.674597 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:31.675539 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:31.676242 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:31.677389 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:31.677833 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:31.677923 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:31.681105 (+ 3182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:31.681109 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:31.681112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:31.681140 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:31.683509 (+ 2369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:31.684059 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:31.684062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:31.684064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:31.684175 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:31.684474 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:31.684477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:31.684479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:31.684712 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:31.684942 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:31.685108 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 23:11:31.685290 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 23:11:31.687166 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 16691
I20260620 23:11:31.697702 17093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:31.687835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50346 (local address 127.10.191.193:42171)
0620 23:11:31.687962 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:31.687966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:31.688318 (+ 352us) server_negotiation.cc:407] Connection header received
0620 23:11:31.688424 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:31.688427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:31.688472 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:31.688542 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:31.689333 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:31.689946 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:31.691111 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:31.691484 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:31.691582 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:31.693319 (+ 1737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:31.693324 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:31.693327 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:31.693352 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:31.695735 (+ 2383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:31.696259 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:31.696262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:31.696264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:31.696378 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:31.696661 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:31.696663 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:31.696665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:31.696926 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:31.697107 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:31.697336 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 23:11:31.697468 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 23:11:31.698211 16814 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:11:31.780396 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 16691
I20260620 23:11:31.796213 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 16825
Jun 20 23:11:31 dist-test-slave-jkbq krb5kdc[16585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.194@KRBTEST.COM
I20260620 23:11:31.810621 17106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:31.797084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50634 (local address 127.10.191.194:40433)
0620 23:11:31.797628 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:31.797635 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:31.797662 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:11:31.797728 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:31.797734 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:31.797941 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:31.798073 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:31.798969 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:31.799949 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:31.801301 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:31.801821 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:31.801917 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:31.805407 (+ 3490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:31.805417 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:31.805468 (+ 51us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:31.805513 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:31.808322 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:31.808878 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:31.808885 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:31.808887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:31.809003 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:31.809305 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:31.809308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:31.809310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:31.809617 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:31.809825 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:31.810069 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 23:11:31.810328 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":303,"threads_started":1}
I20260620 23:11:31.811973 16948 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:11:31.893306 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 16825
I20260620 23:11:31.908886 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 16958
Jun 20 23:11:31 dist-test-slave-jkbq krb5kdc[16585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.195@KRBTEST.COM
I20260620 23:11:31.924826 17108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:31.909846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51430 (local address 127.10.191.195:38991)
0620 23:11:31.910456 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:31.910463 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:31.910492 (+ 29us) server_negotiation.cc:407] Connection header received
0620 23:11:31.910561 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:31.910568 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:31.910831 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:31.911074 (+ 243us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:31.912114 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:31.913138 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:31.914369 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:31.915052 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:31.915194 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:31.919196 (+ 4002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:31.919207 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:31.919221 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:31.919252 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:31.922047 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:31.922667 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:31.922674 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:31.922676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:31.922786 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:31.923371 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:31.923381 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:31.923384 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:31.923738 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:31.923964 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:31.924251 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 23:11:31.924495 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":302,"threads_started":1}
I20260620 23:11:31.926187 17081 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:11:32.009894 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 16958
I20260620 23:11:32.025331 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 16598
I20260620 23:11:32.026580 16660 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:11:32.127748 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 16598
2026-06-20T23:11:32Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3847 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:11:32 dist-test-slave-jkbq krb5kdc[17113](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:11:32 dist-test-slave-jkbq krb5kdc[17113](info): set up 2 sockets
Jun 20 23:11:32 dist-test-slave-jkbq krb5kdc[17113](info): commencing operation
krb5kdc: starting...
W20260620 23:11:33.211490 11007 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Jun 20 23:11:33 dist-test-slave-jkbq krb5kdc[17113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:11:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:11:33Z Disabled control of system clock
WARNING: no policy specified for oryx/127.10.191.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.191.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.191.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.254.keytab.
Entry for principal oryx/127.10.191.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.254.keytab.
I20260620 23:11:33.416577 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:39679
--webserver_interface=127.10.191.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.254.keytab
--principal=oryx/127.10.191.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:33741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:39679
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:11:33.728304 17126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:33.728627 17126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:33.728796 17126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:33.735852 17126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:11:33.735982 17126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:33.736088 17126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:33.736193 17126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:11:33.736275 17126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:11:33.745863 17126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:33741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:39679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.254.keytab
--principal=oryx/127.10.191.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:39679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:33.747249 17126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:33.748589 17126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:33.760528 17131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:33.760447 17134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:33.760493 17132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:33.761691 17126 server_base.cc:1034] running on GCE node
I20260620 23:11:33.762424 17126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:33.763865 17126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:33.765084 17126 hybrid_clock.cc:648] HybridClock initialized: now 1781997093765045 us; error 53 us; skew 500 ppm
I20260620 23:11:33.765415 17126 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:11:33 dist-test-slave-jkbq krb5kdc[17113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997093, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.191.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:33.772786 17126 init.cc:377] Logged in from keytab as oryx/127.10.191.254@KRBTEST.COM (short username oryx)
I20260620 23:11:33.774345 17126 webserver.cc:460] Webserver started at http://127.10.191.254:43223/ using document root <none> and password file <none>
I20260620 23:11:33.775122 17126 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:33.775264 17126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:33.775563 17126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:11:33.778497 17126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "c8e2e42e0532441a97821bf64376c25d"
format_stamp: "Formatted at 2026-06-20 23:11:33 on dist-test-slave-jkbq"
server_key: "9626b99f481c9dee36361d57efc31b5c"
server_key_iv: "9d0cbc1dbb200e1d77f74d30c88e65c8"
server_key_version: "encryptionkey@0"
I20260620 23:11:33.779284 17126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "c8e2e42e0532441a97821bf64376c25d"
format_stamp: "Formatted at 2026-06-20 23:11:33 on dist-test-slave-jkbq"
server_key: "9626b99f481c9dee36361d57efc31b5c"
server_key_iv: "9d0cbc1dbb200e1d77f74d30c88e65c8"
server_key_version: "encryptionkey@0"
I20260620 23:11:33.784395 17126 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 23:11:33.787787 17141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:33.789117 17126 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:11:33.789291 17126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "c8e2e42e0532441a97821bf64376c25d"
format_stamp: "Formatted at 2026-06-20 23:11:33 on dist-test-slave-jkbq"
server_key: "9626b99f481c9dee36361d57efc31b5c"
server_key_iv: "9d0cbc1dbb200e1d77f74d30c88e65c8"
server_key_version: "encryptionkey@0"
I20260620 23:11:33.789484 17126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:33.804675 17126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:33.808480 17126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:33.808758 17126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:33.826824 17126 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:39679
I20260620 23:11:33.826936 17192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:39679 every 8 connection(s)
I20260620 23:11:33.828186 17126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:11:33.831773 17193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:11:33.834771 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 17126
I20260620 23:11:33.835008 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:11:33.835799 11007 external_mini_cluster.cc:1428] Setting key bc0c93b56236b7c41c1c377dc5e93176
I20260620 23:11:33.844408 17193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d: Bootstrap starting.
I20260620 23:11:33.848173 17193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:33.849516 17193 log.cc:824] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d: Log is configured to *not* fsync() on all Append() calls
Jun 20 23:11:33 dist-test-slave-jkbq krb5kdc[17113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM
I20260620 23:11:33.852717 17193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d: No bootstrap required, opened a new log
I20260620 23:11:33.858960 17196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:33.837873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38178 (local address 127.10.191.254:39679)
0620 23:11:33.838619 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:33.838631 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:33.838675 (+ 44us) server_negotiation.cc:407] Connection header received
0620 23:11:33.840348 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:33.840371 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:33.840767 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:33.841114 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:33.842850 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:33.844214 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:33.845930 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:33.846603 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:33.846739 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:33.851723 (+ 4984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:33.851731 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:33.851746 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:33.851785 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:33.854808 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:33.855529 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:33.855544 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:33.855552 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:33.855723 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:33.856127 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:33.856136 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:33.856141 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:33.856641 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:33.857002 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:33.857411 (+ 409us) server_negotiation.cc:299] Negotiation successful
0620 23:11:33.857846 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":366,"threads_started":1}
I20260620 23:11:33.861135 17193 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } }
I20260620 23:11:33.861507 17193 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:33.861629 17193 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8e2e42e0532441a97821bf64376c25d, State: Initialized, Role: FOLLOWER
I20260620 23:11:33.862298 17193 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } }
I20260620 23:11:33.862532 17193 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:11:33.862649 17193 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:11:33.862764 17193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:33.864905 17193 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } }
I20260620 23:11:33.865334 17193 leader_election.cc:302] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8e2e42e0532441a97821bf64376c25d; no voters:
I20260620 23:11:33.865922 17193 leader_election.cc:288] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:11:33.866052 17198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:11:33.867161 17198 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [term 1 LEADER]: Becoming Leader. State: Replica: c8e2e42e0532441a97821bf64376c25d, State: Running, Role: LEADER
I20260620 23:11:33.867676 17198 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } }
I20260620 23:11:33.868108 17193 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:11:33.870179 17199 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8e2e42e0532441a97821bf64376c25d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } }
I20260620 23:11:33.870525 17199 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [sys.catalog]: This master's current role is: LEADER
I20260620 23:11:33.872009 17200 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8e2e42e0532441a97821bf64376c25d. Latest consensus state: current_term: 1 leader_uuid: "c8e2e42e0532441a97821bf64376c25d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } }
I20260620 23:11:33.872287 17200 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [sys.catalog]: This master's current role is: LEADER
I20260620 23:11:33.873533 17205 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:11:33.878805 17205 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:11:33.893565 17205 catalog_manager.cc:1347] Generated new cluster ID: c21dc8b8f31a4eff9c5507da7807c540
I20260620 23:11:33.893781 17205 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:11:33.933168 17205 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:11:33.934438 17205 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:11:33.949602 17205 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d: Generated new TSK 0
I20260620 23:11:33.950452 17205 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.10.191.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.191.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.191.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.193.keytab.
Entry for principal oryx/127.10.191.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.193.keytab.
I20260620 23:11:34.008579 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.193:0
--local_ip_for_outbound_sockets=127.10.191.193
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:39679
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.193.keytab
--principal=oryx/127.10.191.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:33741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:11:34.353269 17219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:34.353531 17219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:34.353717 17219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:34.361307 17219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:34.361409 17219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:34.361629 17219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.193
I20260620 23:11:34.371528 17219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:33741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.193.keytab
--principal=oryx/127.10.191.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:39679
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.193
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:34.373049 17219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:34.374485 17219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:34.387861 17227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:34.387943 17225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:34.387957 17224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:34.388722 17219 server_base.cc:1034] running on GCE node
I20260620 23:11:34.389389 17219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:34.390358 17219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:34.391563 17219 hybrid_clock.cc:648] HybridClock initialized: now 1781997094391479 us; error 73 us; skew 500 ppm
I20260620 23:11:34.391925 17219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:11:34 dist-test-slave-jkbq krb5kdc[17113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997094, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.191.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:34.395269 17219 init.cc:377] Logged in from keytab as oryx/127.10.191.193@KRBTEST.COM (short username oryx)
I20260620 23:11:34.396739 17219 webserver.cc:460] Webserver started at http://127.10.191.193:37087/ using document root <none> and password file <none>
I20260620 23:11:34.397369 17219 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:34.397468 17219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:34.397724 17219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:11:34.400586 17219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-0/data/instance:
uuid: "f3868efeb1fa4926a2fa269532dddbab"
format_stamp: "Formatted at 2026-06-20 23:11:34 on dist-test-slave-jkbq"
server_key: "947167ef544ab69f8889d24103eb920a"
server_key_iv: "229517d80a07e286b00120b7a29e9f4a"
server_key_version: "encryptionkey@0"
I20260620 23:11:34.401312 17219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance:
uuid: "f3868efeb1fa4926a2fa269532dddbab"
format_stamp: "Formatted at 2026-06-20 23:11:34 on dist-test-slave-jkbq"
server_key: "947167ef544ab69f8889d24103eb920a"
server_key_iv: "229517d80a07e286b00120b7a29e9f4a"
server_key_version: "encryptionkey@0"
I20260620 23:11:34.406196 17219 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:11:34.409554 17234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:34.410935 17219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:11:34.411104 17219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-0/wal
uuid: "f3868efeb1fa4926a2fa269532dddbab"
format_stamp: "Formatted at 2026-06-20 23:11:34 on dist-test-slave-jkbq"
server_key: "947167ef544ab69f8889d24103eb920a"
server_key_iv: "229517d80a07e286b00120b7a29e9f4a"
server_key_version: "encryptionkey@0"
I20260620 23:11:34.411304 17219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:34.442744 17219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:34.447286 17219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:34.447580 17219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:34.448540 17219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:11:34.449921 17219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:11:34.450021 17219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:34.450172 17219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:11:34.450253 17219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:34.484043 17219 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.193:40211
I20260620 23:11:34.484090 17346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.193:40211 every 8 connection(s)
I20260620 23:11:34.485282 17219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
I20260620 23:11:34.490540 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 17219
I20260620 23:11:34.490758 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance
I20260620 23:11:34.491394 11007 external_mini_cluster.cc:1428] Setting key be5b4dc57e609cb5a2a3f86b29c1b820
Jun 20 23:11:34 dist-test-slave-jkbq krb5kdc[17113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997094, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.191.193@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM
I20260620 23:11:34.504806 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:34.487834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:53475 (local address 127.10.191.254:39679)
0620 23:11:34.488303 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:34.488307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:34.489166 (+ 859us) server_negotiation.cc:407] Connection header received
0620 23:11:34.491091 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:34.491097 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:34.491186 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:34.491311 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:34.492917 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:34.493737 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:34.495374 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:34.495752 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:34.495885 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:34.499273 (+ 3388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:34.499278 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:34.499281 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:34.499310 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:34.501819 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:34.502496 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:34.502509 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:34.502512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:34.502629 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:34.503130 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:34.503135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:34.503138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:34.503393 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:34.503552 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:34.504230 (+ 678us) server_negotiation.cc:299] Negotiation successful
0620 23:11:34.504448 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":261,"threads_started":1}
I20260620 23:11:34.505781 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:34.488274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.10.191.193:53475)
0620 23:11:34.489001 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:34.489038 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:34.490760 (+ 1722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:34.491548 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:34.491559 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:34.492181 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:34.492740 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:34.492767 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:34.493891 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:34.493898 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:34.495230 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:34.495241 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:34.495968 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:34.495977 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:34.496218 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:34.496920 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:34.496942 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:34.499066 (+ 2124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:34.501954 (+ 2888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:34.501963 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:34.501980 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:34.502376 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:34.502736 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:34.502741 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:34.502752 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:34.503018 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:34.503517 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:34.503527 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:34.503864 (+ 337us) client_negotiation.cc:769] Sending connection context
0620 23:11:34.504133 (+ 269us) client_negotiation.cc:240] Negotiation successful
0620 23:11:34.504633 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":301,"threads_started":1}
I20260620 23:11:34.507370 17347 heartbeater.cc:347] Connected to a master server at 127.10.191.254:39679
I20260620 23:11:34.507699 17347 heartbeater.cc:464] Registering TS with master...
I20260620 23:11:34.508625 17347 heartbeater.cc:511] Master 127.10.191.254:39679 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.10.191.194@KRBTEST.COM; defaulting to no policy
I20260620 23:11:34.510841 17158 ts_manager.cc:194] Registered new tserver with Master: f3868efeb1fa4926a2fa269532dddbab (127.10.191.193:40211)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.191.194@KRBTEST.COM" created.
I20260620 23:11:34.512487 17158 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.191.193@KRBTEST.COM'} at 127.10.191.193:53475
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.191.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.194.keytab.
Entry for principal oryx/127.10.191.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.194.keytab.
I20260620 23:11:34.538120 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.194:0
--local_ip_for_outbound_sockets=127.10.191.194
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:39679
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.194.keytab
--principal=oryx/127.10.191.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:33741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:11:34.858829 17353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:34.859164 17353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:34.859256 17353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:34.866315 17353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:34.866422 17353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:34.866535 17353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.194
I20260620 23:11:34.876415 17353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:33741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.194.keytab
--principal=oryx/127.10.191.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:39679
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.194
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:34.877615 17353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:34.878904 17353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:34.891618 17358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:34.891639 17359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:34.891574 17361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:34.892367 17353 server_base.cc:1034] running on GCE node
I20260620 23:11:34.892949 17353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:34.893891 17353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:34.895119 17353 hybrid_clock.cc:648] HybridClock initialized: now 1781997094895062 us; error 53 us; skew 500 ppm
I20260620 23:11:34.895427 17353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:11:34 dist-test-slave-jkbq krb5kdc[17113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997094, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.191.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:34.898454 17353 init.cc:377] Logged in from keytab as oryx/127.10.191.194@KRBTEST.COM (short username oryx)
I20260620 23:11:34.899839 17353 webserver.cc:460] Webserver started at http://127.10.191.194:43957/ using document root <none> and password file <none>
I20260620 23:11:34.900503 17353 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:34.900629 17353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:34.900921 17353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:11:34.903796 17353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-1/data/instance:
uuid: "ab48ee8fa5634fc59ab80e8a1c3d5361"
format_stamp: "Formatted at 2026-06-20 23:11:34 on dist-test-slave-jkbq"
server_key: "fb948fb8a31e1cc39ac43e97335f4287"
server_key_iv: "45011df2e26d736ac985ae8177ea6d84"
server_key_version: "encryptionkey@0"
I20260620 23:11:34.904567 17353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance:
uuid: "ab48ee8fa5634fc59ab80e8a1c3d5361"
format_stamp: "Formatted at 2026-06-20 23:11:34 on dist-test-slave-jkbq"
server_key: "fb948fb8a31e1cc39ac43e97335f4287"
server_key_iv: "45011df2e26d736ac985ae8177ea6d84"
server_key_version: "encryptionkey@0"
I20260620 23:11:34.909677 17353 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 23:11:34.913134 17368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:34.914520 17353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 23:11:34.914692 17353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-1/wal
uuid: "ab48ee8fa5634fc59ab80e8a1c3d5361"
format_stamp: "Formatted at 2026-06-20 23:11:34 on dist-test-slave-jkbq"
server_key: "fb948fb8a31e1cc39ac43e97335f4287"
server_key_iv: "45011df2e26d736ac985ae8177ea6d84"
server_key_version: "encryptionkey@0"
I20260620 23:11:34.914930 17353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:34.941485 17353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:34.945619 17353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:34.945923 17353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:34.946950 17353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:11:34.948385 17353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:11:34.948457 17353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:34.948613 17353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:11:34.948665 17353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:34.982004 17353 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.194:35513
I20260620 23:11:34.982051 17480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.194:35513 every 8 connection(s)
I20260620 23:11:34.983474 17353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
I20260620 23:11:34.987903 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 17353
I20260620 23:11:34.988070 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance
I20260620 23:11:34.988504 11007 external_mini_cluster.cc:1428] Setting key d1bea592893436e9b0ee14bd197568ad
Jun 20 23:11:34 dist-test-slave-jkbq krb5kdc[17113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997094, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.191.194@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM
I20260620 23:11:35.004308 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:34.986133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:41141 (local address 127.10.191.254:39679)
0620 23:11:34.986313 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:34.986318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:34.987617 (+ 1299us) server_negotiation.cc:407] Connection header received
0620 23:11:34.989663 (+ 2046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:34.989668 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:34.989730 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:34.989823 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:34.991436 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:34.992664 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:34.994290 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:34.995084 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:34.995224 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:34.998931 (+ 3707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:34.998936 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:34.998939 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:34.998969 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:35.001358 (+ 2389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:35.002042 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:35.002048 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:35.002052 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:35.002172 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:35.002609 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:35.002614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:35.002617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:35.002921 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:35.003084 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:35.003753 (+ 669us) server_negotiation.cc:299] Negotiation successful
0620 23:11:35.003965 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 23:11:35.005347 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:34.986613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.10.191.194:41141)
0620 23:11:34.987436 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:34.987480 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:34.989326 (+ 1846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:34.990092 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:34.990102 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:34.990640 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:34.991273 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:34.991290 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:34.992811 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:34.992814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:34.994151 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:34.994167 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:34.995282 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:34.995287 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:34.995467 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:34.996180 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:34.996216 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:34.998692 (+ 2476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:35.001492 (+ 2800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:35.001501 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:35.001518 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:35.001922 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:35.002281 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:35.002287 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:35.002291 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:35.002477 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:35.003054 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:35.003062 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:35.003398 (+ 336us) client_negotiation.cc:769] Sending connection context
0620 23:11:35.003666 (+ 268us) client_negotiation.cc:240] Negotiation successful
0620 23:11:35.004213 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":371,"threads_started":1}
I20260620 23:11:35.007059 17481 heartbeater.cc:347] Connected to a master server at 127.10.191.254:39679
I20260620 23:11:35.007380 17481 heartbeater.cc:464] Registering TS with master...
I20260620 23:11:35.008078 17481 heartbeater.cc:511] Master 127.10.191.254:39679 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.10.191.195@KRBTEST.COM; defaulting to no policy
I20260620 23:11:35.009567 17158 ts_manager.cc:194] Registered new tserver with Master: ab48ee8fa5634fc59ab80e8a1c3d5361 (127.10.191.194:35513)
I20260620 23:11:35.010378 17158 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.191.194@KRBTEST.COM'} at 127.10.191.194:41141
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.191.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.191.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.195.keytab.
Entry for principal oryx/127.10.191.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.195.keytab.
I20260620 23:11:35.034240 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.195:0
--local_ip_for_outbound_sockets=127.10.191.195
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:39679
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.195.keytab
--principal=oryx/127.10.191.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:33741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:11:35.351151 17486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:35.351449 17486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:35.351612 17486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:35.358712 17486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:35.358856 17486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:35.359061 17486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.195
I20260620 23:11:35.368484 17486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:33741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.195.keytab
--principal=oryx/127.10.191.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:39679
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.195
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:35.369853 17486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:35.371214 17486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:35.384078 17492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:35.384136 17494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:35.384160 17491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:35.384974 17486 server_base.cc:1034] running on GCE node
I20260620 23:11:35.385594 17486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:35.386507 17486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:35.387732 17486 hybrid_clock.cc:648] HybridClock initialized: now 1781997095387683 us; error 57 us; skew 500 ppm
I20260620 23:11:35.388049 17486 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:11:35 dist-test-slave-jkbq krb5kdc[17113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997095, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.191.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:35.391330 17486 init.cc:377] Logged in from keytab as oryx/127.10.191.195@KRBTEST.COM (short username oryx)
I20260620 23:11:35.392798 17486 webserver.cc:460] Webserver started at http://127.10.191.195:42537/ using document root <none> and password file <none>
I20260620 23:11:35.393503 17486 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:35.393637 17486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:35.393927 17486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:11:35.396715 17486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-2/data/instance:
uuid: "25499f9be0e240fbbcf6d2864c0b51bf"
format_stamp: "Formatted at 2026-06-20 23:11:35 on dist-test-slave-jkbq"
server_key: "792a62c14c16cdf7af97cee7fa6a3fb0"
server_key_iv: "754acf90b461cc25f1060cf56ee1e121"
server_key_version: "encryptionkey@0"
I20260620 23:11:35.397439 17486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance:
uuid: "25499f9be0e240fbbcf6d2864c0b51bf"
format_stamp: "Formatted at 2026-06-20 23:11:35 on dist-test-slave-jkbq"
server_key: "792a62c14c16cdf7af97cee7fa6a3fb0"
server_key_iv: "754acf90b461cc25f1060cf56ee1e121"
server_key_version: "encryptionkey@0"
I20260620 23:11:35.402668 17486 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 23:11:35.405992 17501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:35.407327 17486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:11:35.407454 17486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-2/wal
uuid: "25499f9be0e240fbbcf6d2864c0b51bf"
format_stamp: "Formatted at 2026-06-20 23:11:35 on dist-test-slave-jkbq"
server_key: "792a62c14c16cdf7af97cee7fa6a3fb0"
server_key_iv: "754acf90b461cc25f1060cf56ee1e121"
server_key_version: "encryptionkey@0"
I20260620 23:11:35.407605 17486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:35.423362 17486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:35.427145 17486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:35.427379 17486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:35.428200 17486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:11:35.429409 17486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:11:35.429476 17486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:35.429557 17486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:11:35.429600 17486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:35.462390 17486 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.195:46823
I20260620 23:11:35.462479 17613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.195:46823 every 8 connection(s)
I20260620 23:11:35.463696 17486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
I20260620 23:11:35.472909 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 17486
I20260620 23:11:35.473093 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance
I20260620 23:11:35.473541 11007 external_mini_cluster.cc:1428] Setting key 530048eb663ce7dd85bde4cdd040159a
Jun 20 23:11:35 dist-test-slave-jkbq krb5kdc[17113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997095, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.191.195@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM
I20260620 23:11:35.484339 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:35.466217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:57813 (local address 127.10.191.254:39679)
0620 23:11:35.466384 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:35.466388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:35.467547 (+ 1159us) server_negotiation.cc:407] Connection header received
0620 23:11:35.470099 (+ 2552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:35.470105 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:35.470195 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:35.470310 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:35.472133 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:35.472959 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:35.475167 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:35.475710 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:35.475806 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:35.479214 (+ 3408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:35.479219 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:35.479222 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:35.479252 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:35.481548 (+ 2296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:35.482177 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:35.482181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:35.482183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:35.482289 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:35.482654 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:35.482657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:35.482659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:35.482958 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:35.483062 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:35.483814 (+ 752us) server_negotiation.cc:299] Negotiation successful
0620 23:11:35.484027 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 23:11:35.485247 17616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:35.466635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.10.191.195:57813)
0620 23:11:35.467360 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:35.467396 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:35.469782 (+ 2386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:35.470661 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:35.470675 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:35.471420 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:35.471971 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:35.471988 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:35.473224 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:35.473230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:35.475037 (+ 1807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:35.475049 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:35.475927 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:35.475934 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:35.476207 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:35.476925 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:35.476947 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:35.479038 (+ 2091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:35.481673 (+ 2635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:35.481679 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:35.481690 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:35.482074 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:35.482384 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:35.482387 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:35.482389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:35.482548 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:35.483056 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:35.483061 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:35.483360 (+ 299us) client_negotiation.cc:769] Sending connection context
0620 23:11:35.483613 (+ 253us) client_negotiation.cc:240] Negotiation successful
0620 23:11:35.484095 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":323,"threads_started":1}
I20260620 23:11:35.486541 17614 heartbeater.cc:347] Connected to a master server at 127.10.191.254:39679
I20260620 23:11:35.486896 17614 heartbeater.cc:464] Registering TS with master...
I20260620 23:11:35.487684 17614 heartbeater.cc:511] Master 127.10.191.254:39679 requested a full tablet report, sending...
I20260620 23:11:35.489147 17158 ts_manager.cc:194] Registered new tserver with Master: 25499f9be0e240fbbcf6d2864c0b51bf (127.10.191.195:46823)
I20260620 23:11:35.489929 17158 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.191.195@KRBTEST.COM'} at 127.10.191.195:57813
I20260620 23:11:35.500876 11007 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:11:35.516141 17347 heartbeater.cc:503] Master 127.10.191.254:39679 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.10.191.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.191.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.191.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.253.keytab.
Entry for principal oryx/127.10.191.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.253.keytab.
I20260620 23:11:35.545876 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.253:33657
--webserver_interface=127.10.191.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.253.keytab
--principal=oryx/127.10.191.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:33741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:39679,127.10.191.253:33657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:11:35.859150 17619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:35.859496 17619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:35.859648 17619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:35.866981 17619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:11:35.867125 17619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:35.867233 17619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:35.867341 17619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:11:35.867415 17619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:11:35.877641 17619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:33741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:39679,127.10.191.253:33657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.253.keytab
--principal=oryx/127.10.191.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.253:33657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.191.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
W20260620 23:11:35.878034 17619 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.10.191.254:39679,127.10.191.253:33657'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 23:11:35.879146 17619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:35.880546 17619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:35.892542 17627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:35.892542 17625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:35.892648 17624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:35.893370 17619 server_base.cc:1034] running on GCE node
I20260620 23:11:35.894193 17619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:35.895694 17619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:35.896953 17619 hybrid_clock.cc:648] HybridClock initialized: now 1781997095896884 us; error 47 us; skew 500 ppm
I20260620 23:11:35.897312 17619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:11:35 dist-test-slave-jkbq krb5kdc[17113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997095, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.191.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:35.900435 17619 init.cc:377] Logged in from keytab as oryx/127.10.191.253@KRBTEST.COM (short username oryx)
I20260620 23:11:35.901860 17619 webserver.cc:460] Webserver started at http://127.10.191.253:45883/ using document root <none> and password file <none>
I20260620 23:11:35.902592 17619 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:35.902729 17619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:35.903105 17619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:11:35.906016 17619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-1/data/instance:
uuid: "6db99becd3324a5aaaf1eaf5985ae3f0"
format_stamp: "Formatted at 2026-06-20 23:11:35 on dist-test-slave-jkbq"
server_key: "b5453b995fac4d1c03abd74722cd48d8"
server_key_iv: "3719713cd23f8b51f38eca07f16f482e"
server_key_version: "encryptionkey@0"
I20260620 23:11:35.906790 17619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-1/wal/instance:
uuid: "6db99becd3324a5aaaf1eaf5985ae3f0"
format_stamp: "Formatted at 2026-06-20 23:11:35 on dist-test-slave-jkbq"
server_key: "b5453b995fac4d1c03abd74722cd48d8"
server_key_iv: "3719713cd23f8b51f38eca07f16f482e"
server_key_version: "encryptionkey@0"
I20260620 23:11:35.912114 17619 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 23:11:35.915547 17634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:35.916901 17619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:11:35.917081 17619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-1/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-1/wal
uuid: "6db99becd3324a5aaaf1eaf5985ae3f0"
format_stamp: "Formatted at 2026-06-20 23:11:35 on dist-test-slave-jkbq"
server_key: "b5453b995fac4d1c03abd74722cd48d8"
server_key_iv: "3719713cd23f8b51f38eca07f16f482e"
server_key_version: "encryptionkey@0"
I20260620 23:11:35.917307 17619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:35.942095 17619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:35.946269 17619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:35.946545 17619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:35.964622 17619 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.253:33657
I20260620 23:11:35.964663 17685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.253:33657 every 8 connection(s)
I20260620 23:11:35.966018 17619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-1/data/info.pb
I20260620 23:11:35.969842 17686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:11:35.974642 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 17619
I20260620 23:11:35.977097 17686 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:11:35 dist-test-slave-jkbq krb5kdc[17113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997095, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.191.253@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM
Jun 20 23:11:35 dist-test-slave-jkbq krb5kdc[17113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.191.253@KRBTEST.COM
I20260620 23:11:35.996047 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:35.979669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38190 (local address 127.10.191.254:39679)
0620 23:11:35.979820 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:35.979824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:35.980527 (+ 703us) server_negotiation.cc:407] Connection header received
0620 23:11:35.980768 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:35.980774 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:35.980857 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:35.980934 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:35.982748 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:35.983591 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:35.985283 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:35.985782 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:35.985913 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:35.989916 (+ 4003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:35.989922 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:35.989928 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:35.989969 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:35.992750 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:35.993726 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:35.993731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:35.993735 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:35.993864 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:35.994306 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:35.994314 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:35.994317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:35.994618 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:35.994802 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:35.995544 (+ 742us) server_negotiation.cc:299] Negotiation successful
0620 23:11:35.995786 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 23:11:35.997231 17690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:35.979763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.0.0.1:38190)
0620 23:11:35.980391 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:35.980422 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:35.980597 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:35.981208 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:35.981216 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:35.982059 (+ 843us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:35.982566 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:35.982593 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:35.983739 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:35.983744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:35.985137 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:35.985148 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:35.985998 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:35.986005 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:35.986214 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:35.986924 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:35.986955 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:35.989720 (+ 2765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:35.992922 (+ 3202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:35.992938 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:35.992949 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:35.993590 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:35.994000 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:35.994006 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:35.994009 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:35.994195 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:35.994775 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:35.994781 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:35.995140 (+ 359us) client_negotiation.cc:769] Sending connection context
0620 23:11:35.995445 (+ 305us) client_negotiation.cc:240] Negotiation successful
0620 23:11:35.995869 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":317,"threads_started":1}
I20260620 23:11:35.997388 17689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:35.976898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42414 (local address 127.10.191.253:33657)
0620 23:11:35.977761 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:35.977772 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:35.977811 (+ 39us) server_negotiation.cc:407] Connection header received
0620 23:11:35.979868 (+ 2057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:35.979897 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:35.980427 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:35.980781 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:35.982561 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:35.984412 (+ 1851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:35.986547 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:35.987218 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:35.987372 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:35.991631 (+ 4259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:35.991643 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:35.991659 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:35.991702 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:35.994579 (+ 2877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:35.995298 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:35.995305 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:35.995308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:35.995447 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:35.995844 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:35.995849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:35.995851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:35.996236 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:35.996512 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:35.996795 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 23:11:35.997002 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":450,"threads_started":1}
I20260620 23:11:35.999266 17686 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:11:36 dist-test-slave-jkbq krb5kdc[17113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997095, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.191.253@KRBTEST.COM for oryx/127.10.191.253@KRBTEST.COM
I20260620 23:11:36.011113 17690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:35.999970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:33657 (local address 127.0.0.1:42420)
0620 23:11:36.000153 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:36.000173 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:36.000273 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:36.000566 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:36.000569 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:36.000870 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:36.000993 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:36.001000 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.001895 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.001898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:36.002808 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:36.002819 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.003517 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.003521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:36.003679 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:36.004173 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:36.004189 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:36.006209 (+ 2020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:36.009027 (+ 2818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:36.009030 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:36.009033 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:36.009408 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:36.009731 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:36.009734 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:36.009736 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:36.009851 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:36.010288 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:36.010291 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:36.010406 (+ 115us) client_negotiation.cc:769] Sending connection context
0620 23:11:36.010507 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 23:11:36.010753 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 23:11:36.011191 17689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:36.000150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42420 (local address 127.10.191.253:33657)
0620 23:11:36.000293 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:36.000297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:36.000311 (+ 14us) server_negotiation.cc:407] Connection header received
0620 23:11:36.000372 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:36.000377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:36.000442 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:36.000542 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:36.001104 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.001769 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.002994 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.003382 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.003477 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:36.006375 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:36.006379 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:36.006382 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:36.006409 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:36.008906 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:36.009520 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:36.009523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:36.009526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:36.009632 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:36.009938 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:36.009940 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:36.009942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:36.010198 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:36.010298 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:36.010623 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 23:11:36.010945 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":49}
I20260620 23:11:36.012872 17481 heartbeater.cc:503] Master 127.10.191.254:39679 was elected leader, sending a full tablet report...
I20260620 23:11:36.016175 17686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0: Bootstrap starting.
I20260620 23:11:36.019387 17686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:36.020403 17686 log.cc:824] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0: Log is configured to *not* fsync() on all Append() calls
I20260620 23:11:36.023111 17686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0: No bootstrap required, opened a new log
I20260620 23:11:36.029789 17686 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } }
I20260620 23:11:36.030068 17686 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:36.030151 17686 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6db99becd3324a5aaaf1eaf5985ae3f0, State: Initialized, Role: FOLLOWER
I20260620 23:11:36.030615 17686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } }
I20260620 23:11:36.031419 17692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } } }
I20260620 23:11:36.031694 17692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:11:36.032375 17686 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 23:11:36.040422 17703 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:11:36.040562 17703 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:11:36.043236 17619 master_runner.cc:428] Detected that this master 6db99becd3324a5aaaf1eaf5985ae3f0 is joining an existing cluster
I20260620 23:11:36.043356 17619 master_runner.cc:432] Initiating AddMaster RPC to add 127.10.191.253:33657
I20260620 23:11:36.063974 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:36.047015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38192 (local address 127.10.191.254:39679)
0620 23:11:36.047213 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:36.047219 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:36.047997 (+ 778us) server_negotiation.cc:407] Connection header received
0620 23:11:36.048269 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:36.048275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:36.048359 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:36.048435 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:36.050262 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.051284 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.053355 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.054108 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.054219 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:36.057380 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:36.057389 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:36.057397 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:36.057442 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:36.061230 (+ 3788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:36.061921 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:36.061929 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:36.061933 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:36.062074 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:36.062461 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:36.062469 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:36.062473 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:36.062765 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:36.062950 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:36.063394 (+ 444us) server_negotiation.cc:299] Negotiation successful
0620 23:11:36.063597 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 23:11:36.064090 17709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:36.047249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.0.0.1:38192)
0620 23:11:36.047898 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:36.047917 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:36.048114 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:36.048688 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:36.048694 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:36.049595 (+ 901us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:36.050084 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:36.050101 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.051403 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.051411 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:36.053186 (+ 1775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:36.053200 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.054520 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.054529 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:36.054831 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:36.055717 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:36.055748 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:36.057170 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:36.061368 (+ 4198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:36.061376 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:36.061380 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:36.061789 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:36.062184 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:36.062197 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:36.062201 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:36.062352 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:36.062908 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:36.062913 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:36.063109 (+ 196us) client_negotiation.cc:769] Sending connection context
0620 23:11:36.063273 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 23:11:36.063587 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"mutex_wait_us":75,"spinlock_wait_cycles":1408,"thread_start_us":345,"threads_started":1}
I20260620 23:11:36.064581 17689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:36.048182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42422 (local address 127.10.191.253:33657)
0620 23:11:36.048367 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:36.048373 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:36.048807 (+ 434us) server_negotiation.cc:407] Connection header received
0620 23:11:36.048985 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:36.048990 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:36.049056 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:36.049155 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:36.050323 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.051434 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.053473 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.054209 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.054307 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:36.057334 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:36.057345 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:36.057350 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:36.057382 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:36.061367 (+ 3985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:36.062299 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:36.062304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:36.062308 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:36.062455 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:36.062922 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:36.062929 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:36.062934 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:36.063302 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:36.063439 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:36.063970 (+ 531us) server_negotiation.cc:299] Negotiation successful
0620 23:11:36.064202 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38,"spinlock_wait_cycles":1408}
I20260620 23:11:36.066398 17710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:36.047997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:33657 (local address 127.0.0.1:42422)
0620 23:11:36.048709 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:36.048722 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:36.048870 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:36.049167 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:36.049171 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:36.049762 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:36.050181 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:36.050211 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.051676 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.051686 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:36.053330 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:36.053343 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.054589 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.054597 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:36.054840 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:36.055836 (+ 996us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:36.055859 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:36.057148 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:36.061526 (+ 4378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:36.061530 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:36.061534 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:36.062170 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:36.062598 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:36.062603 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:36.062606 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:36.062759 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:36.063541 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:36.063546 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:36.063681 (+ 135us) client_negotiation.cc:769] Sending connection context
0620 23:11:36.065653 (+ 1972us) client_negotiation.cc:240] Negotiation successful
0620 23:11:36.065925 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":108,"spinlock_wait_cycles":3968,"thread_start_us":435,"threads_started":1}
WARNING: no policy specified for oryx/127.10.191.252@KRBTEST.COM; defaulting to no policy
Jun 20 23:11:36 dist-test-slave-jkbq krb5kdc[17113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997093, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.191.254@KRBTEST.COM for oryx/127.10.191.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.191.252@KRBTEST.COM" created.
I20260620 23:11:36.088440 17712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:36.070786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:33657 (local address 127.0.0.1:42424)
0620 23:11:36.071442 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:36.071481 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:36.071697 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:36.072211 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:36.072222 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:36.073046 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:36.073338 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:36.073353 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.074774 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.074782 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:36.076281 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:36.076293 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.077329 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.077337 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:36.077592 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:36.078320 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:36.078343 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:36.081261 (+ 2918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:36.085283 (+ 4022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:36.085296 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:36.085304 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:36.085872 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:36.086307 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:36.086312 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:36.086315 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:36.086473 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:36.087249 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:36.087258 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:36.087460 (+ 202us) client_negotiation.cc:769] Sending connection context
0620 23:11:36.087668 (+ 208us) client_negotiation.cc:240] Negotiation successful
0620 23:11:36.087999 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":284,"threads_started":1}
I20260620 23:11:36.088889 17689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:36.070720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42424 (local address 127.10.191.253:33657)
0620 23:11:36.070923 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:36.070930 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:36.071558 (+ 628us) server_negotiation.cc:407] Connection header received
0620 23:11:36.071906 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:36.071911 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:36.072043 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:36.072153 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:36.073490 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.074630 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.076436 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.077141 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.077277 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:36.081486 (+ 4209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:36.081495 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:36.081502 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:36.081542 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:36.085133 (+ 3591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:36.086018 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:36.086026 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:36.086032 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:36.086187 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:36.086620 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:36.086625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:36.086636 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:36.087027 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:36.087177 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:36.088175 (+ 998us) server_negotiation.cc:299] Negotiation successful
0620 23:11:36.088429 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 23:11:36.091025 17158 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.10.191.253:33657: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0"
member_type: NON_VOTER
last_known_addr {
host: "127.10.191.253"
port: 33657
}
attrs {
promote: true
}
}
dest_uuid: "c8e2e42e0532441a97821bf64376c25d"
cas_config_opid_index: -1
I20260620 23:11:36.092193 17158 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: NON_VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: true } }
I20260620 23:11:36.103214 17661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:36.107506 17661 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [term 1 FOLLOWER]: Refusing update from remote peer c8e2e42e0532441a97821bf64376c25d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 23:11:36.107309 17715 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 6db99becd3324a5aaaf1eaf5985ae3f0 (127.10.191.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: NON_VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: true } } }
I20260620 23:11:36.108737 17716 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: NON_VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:11:36.112157 17715 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.10.191.253:33657
I20260620 23:11:36.113051 17619 master.cc:537] Master@127.10.191.253:33657 shutting down...
I20260620 23:11:36.112994 17716 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c8e2e42e0532441a97821bf64376c25d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: NON_VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: true } } }
I20260620 23:11:36.113616 17716 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.191.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.252.keytab.
Entry for principal oryx/127.10.191.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.252.keytab.
I20260620 23:11:36.119513 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.252:44225
--webserver_interface=127.10.191.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.252.keytab
--principal=oryx/127.10.191.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:33741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:39679,127.10.191.253:33657,127.10.191.252:44225
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:11:36.141614 17661 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 6db99becd3324a5aaaf1eaf5985ae3f0 (127.10.191.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: NON_VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: true } } }
I20260620 23:11:36.155669 17725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8e2e42e0532441a97821bf64376c25d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: NON_VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: true } } }
I20260620 23:11:36.156117 17692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8e2e42e0532441a97821bf64376c25d. Latest consensus state: current_term: 1 leader_uuid: "c8e2e42e0532441a97821bf64376c25d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: NON_VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: true } } }
I20260620 23:11:36.156363 17692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [sys.catalog]: This master's current role is: LEARNER
I20260620 23:11:36.156313 17725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [sys.catalog]: This master's current role is: LEARNER
I20260620 23:11:36.156611 17721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8e2e42e0532441a97821bf64376c25d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: NON_VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: true } } }
I20260620 23:11:36.156949 17721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [sys.catalog]: This master's current role is: LEARNER
I20260620 23:11:36.159488 17619 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [term 1 LEARNER]: Raft consensus shutting down.
I20260620 23:11:36.159976 17619 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [term 1 LEARNER]: Raft consensus is shut down!
I20260620 23:11:36.160133 17619 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0: stopping tablet replica
I20260620 23:11:36.167626 17619 master.cc:552] Master@127.10.191.253:33657 shutdown complete.
I20260620 23:11:36.167711 17619 master_runner.cc:305] Clearing existing system tablet
I20260620 23:11:36.168061 17619 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:36.168217 17619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:36.170588 17619 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260620 23:11:36.172073 17727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:36.172369 17619 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:36.172672 17619 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:11:36.172755 17619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-1/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-1/wal
uuid: "6db99becd3324a5aaaf1eaf5985ae3f0"
format_stamp: "Formatted at 2026-06-20 23:11:35 on dist-test-slave-jkbq"
server_key: "b5453b995fac4d1c03abd74722cd48d8"
server_key_iv: "3719713cd23f8b51f38eca07f16f482e"
server_key_version: "encryptionkey@0"
I20260620 23:11:36.175982 17619 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:11:36.180621 17619 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 23:11:36.180806 17619 log.cc:1196] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 23:11:36.181216 17619 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0: Deleting consensus metadata
I20260620 23:11:36.181474 17619 master_runner.cc:315] Copying system tablet from 127.10.191.254:39679
I20260620 23:11:36.191881 17619 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0: tablet copy: Beginning tablet copy session from remote peer at address 127.10.191.254:39679
I20260620 23:11:36.204422 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:36.192651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38202 (local address 127.10.191.254:39679)
0620 23:11:36.192852 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:36.192859 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:36.193358 (+ 499us) server_negotiation.cc:407] Connection header received
0620 23:11:36.193612 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:36.193618 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:36.193707 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:36.193834 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:36.194766 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.195534 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.196803 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.197246 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.197381 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:36.199233 (+ 1852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:36.199239 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:36.199245 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:36.199282 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:36.201847 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:36.202470 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:36.202476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:36.202480 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:36.202615 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:36.203034 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:36.203039 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:36.203043 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:36.203308 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:36.203474 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:36.203837 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 23:11:36.204049 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 23:11:36.204502 17733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:36.192617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.0.0.1:38202)
0620 23:11:36.193236 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:36.193256 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:36.193428 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:36.193890 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:36.193894 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:36.194320 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:36.194622 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:36.194632 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.195683 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.195688 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:36.196656 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:36.196667 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.197392 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.197399 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:36.197585 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:36.198213 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:36.198230 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:36.199051 (+ 821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:36.201980 (+ 2929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:36.201987 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:36.201991 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:36.202333 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:36.202729 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:36.202734 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:36.202738 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:36.202926 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:36.203426 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:36.203431 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:36.203607 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 23:11:36.203749 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 23:11:36.204047 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":369,"threads_started":1}
I20260620 23:11:36.205586 17178 tablet_copy_service.cc:140] P c8e2e42e0532441a97821bf64376c25d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 6db99becd3324a5aaaf1eaf5985ae3f0 ({username='oryx', principal='oryx/127.10.191.253@KRBTEST.COM'} at 127.0.0.1:38202)
I20260620 23:11:36.205835 17178 tablet_copy_service.cc:161] P c8e2e42e0532441a97821bf64376c25d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 6db99becd3324a5aaaf1eaf5985ae3f0 at {username='oryx', principal='oryx/127.10.191.253@KRBTEST.COM'} at 127.0.0.1:38202: session id = 6db99becd3324a5aaaf1eaf5985ae3f0-00000000000000000000000000000000
I20260620 23:11:36.208823 17178 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 23:11:36.210728 17619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 23:11:36.215687 17619 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0: tablet copy: Starting download of 0 data blocks...
I20260620 23:11:36.215981 17619 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0: tablet copy: Starting download of 1 WAL segments...
I20260620 23:11:36.218312 17619 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 23:11:36.221650 17178 tablet_copy_service.cc:342] P c8e2e42e0532441a97821bf64376c25d: Request end of tablet copy session 6db99becd3324a5aaaf1eaf5985ae3f0-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.10.191.253@KRBTEST.COM'} at 127.0.0.1:38202
I20260620 23:11:36.221829 17178 tablet_copy_service.cc:433] P c8e2e42e0532441a97821bf64376c25d: ending tablet copy session 6db99becd3324a5aaaf1eaf5985ae3f0-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 6db99becd3324a5aaaf1eaf5985ae3f0
I20260620 23:11:36.231616 17619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:36.240135 17734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:36.247665 17735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:36.250723 17619 server_base.cc:1034] running on GCE node
W20260620 23:11:36.251065 17738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:36.251684 17619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:36.252494 17619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:36.253769 17619 hybrid_clock.cc:648] HybridClock initialized: now 1781997096253704 us; error 43 us; skew 500 ppm
I20260620 23:11:36.254004 17619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:11:36 dist-test-slave-jkbq krb5kdc[17113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997096, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.191.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:36.256049 17619 init.cc:377] Logged in from keytab as oryx/127.10.191.253@KRBTEST.COM (short username oryx)
I20260620 23:11:36.257283 17619 webserver.cc:460] Webserver started at http://127.10.191.253:37189/ using document root <none> and password file <none>
I20260620 23:11:36.257617 17619 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:36.257707 17619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:36.260083 17619 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 23:11:36.261683 17744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:36.262084 17619 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:11:36.262217 17619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-1/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-1/wal
uuid: "6db99becd3324a5aaaf1eaf5985ae3f0"
format_stamp: "Formatted at 2026-06-20 23:11:35 on dist-test-slave-jkbq"
server_key: "b5453b995fac4d1c03abd74722cd48d8"
server_key_iv: "3719713cd23f8b51f38eca07f16f482e"
server_key_version: "encryptionkey@0"
I20260620 23:11:36.262387 17619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:36.271013 17619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:36.274325 17619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:36.320595 17619 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.253:33657
I20260620 23:11:36.320595 17795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.253:33657 every 8 connection(s)
I20260620 23:11:36.321588 17619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-1/data/info.pb
I20260620 23:11:36.329213 17796 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:11:36.334645 17796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0: Bootstrap starting.
I20260620 23:11:36.345564 17796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:11:36.346035 17796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0: Bootstrap complete.
I20260620 23:11:36.346921 17796 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: NON_VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: true } }
I20260620 23:11:36.347157 17796 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6db99becd3324a5aaaf1eaf5985ae3f0, State: Initialized, Role: LEARNER
I20260620 23:11:36.347425 17796 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: NON_VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: true } }
I20260620 23:11:36.348539 17799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: NON_VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: true } } }
I20260620 23:11:36.348855 17796 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:11:36.348996 17799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [sys.catalog]: This master's current role is: LEARNER
I20260620 23:11:36.358718 17810 catalog_manager.cc:1259] Loaded cluster ID: c21dc8b8f31a4eff9c5507da7807c540
I20260620 23:11:36.358841 17810 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0: loading cluster ID for follower catalog manager: success
I20260620 23:11:36.361742 17810 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0: acquiring CA information for follower catalog manager: success
I20260620 23:11:36.363966 17810 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:11:36.492357 17614 heartbeater.cc:503] Master 127.10.191.254:39679 was elected leader, sending a full tablet report...
W20260620 23:11:36.543803 17719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:36.544142 17719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:36.544296 17719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:36.551729 17719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:11:36.551875 17719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:36.551975 17719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:36.552070 17719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:11:36.552135 17719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:11:36.562667 17719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:33741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:39679,127.10.191.253:33657,127.10.191.252:44225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.252.keytab
--principal=oryx/127.10.191.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.252:44225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.191.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:36.564273 17719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:36.565665 17719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:36.578015 17815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:36.578099 17816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:36.578063 17818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:36.578914 17719 server_base.cc:1034] running on GCE node
I20260620 23:11:36.579706 17719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:36.581269 17719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:36.582505 17719 hybrid_clock.cc:648] HybridClock initialized: now 1781997096582464 us; error 70 us; skew 500 ppm
I20260620 23:11:36.582927 17719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:11:36 dist-test-slave-jkbq krb5kdc[17113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997096, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.191.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:36.586213 17719 init.cc:377] Logged in from keytab as oryx/127.10.191.252@KRBTEST.COM (short username oryx)
I20260620 23:11:36.587782 17719 webserver.cc:460] Webserver started at http://127.10.191.252:33255/ using document root <none> and password file <none>
I20260620 23:11:36.588501 17719 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:36.588653 17719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:36.588964 17719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:11:36.592082 17719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-2/data/instance:
uuid: "806cc6e3c2fa46b7a43e12917c49150c"
format_stamp: "Formatted at 2026-06-20 23:11:36 on dist-test-slave-jkbq"
server_key: "0e718311e3f81fb2aad8f68c994b427a"
server_key_iv: "bc9a14ea3037721246ab9bdb830153dd"
server_key_version: "encryptionkey@0"
I20260620 23:11:36.592906 17719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-2/wal/instance:
uuid: "806cc6e3c2fa46b7a43e12917c49150c"
format_stamp: "Formatted at 2026-06-20 23:11:36 on dist-test-slave-jkbq"
server_key: "0e718311e3f81fb2aad8f68c994b427a"
server_key_iv: "bc9a14ea3037721246ab9bdb830153dd"
server_key_version: "encryptionkey@0"
I20260620 23:11:36.598014 17719 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 23:11:36.602478 17825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:36.603929 17719 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:11:36.604116 17719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-2/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-2/wal
uuid: "806cc6e3c2fa46b7a43e12917c49150c"
format_stamp: "Formatted at 2026-06-20 23:11:36 on dist-test-slave-jkbq"
server_key: "0e718311e3f81fb2aad8f68c994b427a"
server_key_iv: "bc9a14ea3037721246ab9bdb830153dd"
server_key_version: "encryptionkey@0"
I20260620 23:11:36.604323 17719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:36.652660 17719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:36.657719 17719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:36.658020 17719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:36.676820 17876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.252:44225 every 8 connection(s)
I20260620 23:11:36.676828 17719 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.252:44225
I20260620 23:11:36.678359 17719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-2/data/info.pb
I20260620 23:11:36.682186 17877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:11:36.683564 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 17719
I20260620 23:11:36.691969 17877 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:11:36 dist-test-slave-jkbq krb5kdc[17113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.191.252@KRBTEST.COM
Jun 20 23:11:36 dist-test-slave-jkbq krb5kdc[17113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997096, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.191.252@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM
I20260620 23:11:36.708277 17880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:36.685800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38310 (local address 127.10.191.252:44225)
0620 23:11:36.686571 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:36.686587 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:36.686649 (+ 62us) server_negotiation.cc:407] Connection header received
0620 23:11:36.688774 (+ 2125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:36.688803 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:36.689343 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:36.689816 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:36.691289 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.693263 (+ 1974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.695272 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.696011 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.696225 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:36.700592 (+ 4367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:36.700602 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:36.700613 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:36.700648 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:36.704011 (+ 3363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:36.704716 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:36.704722 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:36.704729 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:36.704872 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:36.705250 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:36.705260 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:36.705262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:36.705735 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:36.706036 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:36.706482 (+ 446us) server_negotiation.cc:299] Negotiation successful
0620 23:11:36.706918 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":315,"threads_started":1}
I20260620 23:11:36.709784 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:36.695078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38218 (local address 127.10.191.254:39679)
0620 23:11:36.695273 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:36.695279 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:36.696011 (+ 732us) server_negotiation.cc:407] Connection header received
0620 23:11:36.696220 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:36.696225 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:36.696288 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:36.696342 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:36.697693 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.698393 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.699941 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.700332 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.700498 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:36.704137 (+ 3639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:36.704143 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:36.704146 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:36.704188 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:36.706663 (+ 2475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:36.707584 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:36.707589 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:36.707591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:36.707750 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:36.708146 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:36.708156 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:36.708158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:36.708468 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:36.708609 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:36.709241 (+ 632us) server_negotiation.cc:299] Negotiation successful
0620 23:11:36.709453 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 23:11:36.709851 17881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:36.695047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.0.0.1:38218)
0620 23:11:36.695857 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:36.695908 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:36.696051 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:36.696530 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:36.696535 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:36.697289 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:36.697526 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:36.697534 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.698543 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.698549 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:36.699779 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:36.699792 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.700593 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.700599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:36.700774 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:36.701443 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:36.701470 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:36.703945 (+ 2475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:36.706853 (+ 2908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:36.706884 (+ 31us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:36.706892 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:36.707438 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:36.707874 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:36.707879 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:36.707883 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:36.708028 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:36.708704 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:36.708714 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:36.708903 (+ 189us) client_negotiation.cc:769] Sending connection context
0620 23:11:36.709125 (+ 222us) client_negotiation.cc:240] Negotiation successful
0620 23:11:36.709428 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":417,"threads_started":1}
I20260620 23:11:36.712230 17877 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:11:36 dist-test-slave-jkbq krb5kdc[17113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997096, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.191.252@KRBTEST.COM for oryx/127.10.191.253@KRBTEST.COM
I20260620 23:11:36.724938 17881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:36.712819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:33657 (local address 127.0.0.1:42426)
0620 23:11:36.713010 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:36.713025 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:36.713099 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:36.713806 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:36.713810 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:36.714184 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:36.714335 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:36.714343 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.715708 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.715714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:36.716670 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:36.716681 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.717313 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.717317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:36.717460 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:36.717992 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:36.718008 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:36.720069 (+ 2061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:36.723027 (+ 2958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:36.723031 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:36.723033 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:36.723396 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:36.723730 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:36.723733 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:36.723735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:36.723865 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:36.724295 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:36.724298 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:36.724406 (+ 108us) client_negotiation.cc:769] Sending connection context
0620 23:11:36.724509 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 23:11:36.724723 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 23:11:36.725131 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:36.713021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42426 (local address 127.10.191.253:33657)
0620 23:11:36.713498 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:36.713503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:36.713517 (+ 14us) server_negotiation.cc:407] Connection header received
0620 23:11:36.713562 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:36.713566 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:36.713622 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:36.713778 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:36.714503 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.715576 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.716795 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.717180 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.717282 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:36.720276 (+ 2994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:36.720281 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:36.720284 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:36.720315 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:36.722812 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:36.723491 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:36.723495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:36.723498 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:36.723617 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:36.723951 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:36.723955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:36.723957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:36.724206 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:36.724321 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:36.724629 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 23:11:36.724798 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":240,"threads_started":1}
I20260620 23:11:36.726089 17877 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.191.252" port: 44225 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:11:36.728166 17883 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8e2e42e0532441a97821bf64376c25d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: NON_VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: true } } }
I20260620 23:11:36.728529 17883 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [sys.catalog]: This master's current role is: LEADER
Jun 20 23:11:36 dist-test-slave-jkbq krb5kdc[17113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997096, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.191.252@KRBTEST.COM for oryx/127.10.191.252@KRBTEST.COM
I20260620 23:11:36.741053 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:36.728605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42430 (local address 127.10.191.253:33657)
0620 23:11:36.728812 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:36.728820 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:36.735396 (+ 6576us) server_negotiation.cc:407] Connection header received
0620 23:11:36.735579 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:36.735585 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:36.735698 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:36.735817 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:11:36.736243 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.737033 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.738719 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.740085 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.740293 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:36.740504 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:36.740691 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 23:11:36.740818 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 23:11:36.741124 17886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:36.728230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:33657 (local address 127.0.0.1:42430)
0620 23:11:36.735270 (+ 7040us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:36.735291 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:36.735435 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:36.735853 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:36.735857 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:36.735880 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:11:36.736101 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:36.736110 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.737169 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.737173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:36.738558 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:36.738571 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.740278 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.740284 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:36.740485 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:36.740542 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 23:11:36.740657 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 23:11:36.740801 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6889,"thread_start_us":6773,"threads_started":1}
I20260620 23:11:36.742389 17881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:36.726735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:44225 (local address 127.0.0.1:38314)
0620 23:11:36.726967 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:36.726989 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:36.727100 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:36.727436 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:36.727439 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:36.727865 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:36.728053 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:36.728065 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.729816 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.729823 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:36.731625 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:36.731639 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.732932 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.732940 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:36.733244 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:36.733983 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:36.734015 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:36.736670 (+ 2655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:36.740050 (+ 3380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:36.740053 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:36.740056 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:36.740386 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:36.740738 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:36.740744 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:36.740752 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:36.740883 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:36.741506 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:36.741532 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:36.741708 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 23:11:36.741850 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 23:11:36.742054 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 23:11:36.742432 17799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8e2e42e0532441a97821bf64376c25d. Latest consensus state: current_term: 1 leader_uuid: "c8e2e42e0532441a97821bf64376c25d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: NON_VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: true } } }
I20260620 23:11:36.742723 17799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [sys.catalog]: This master's current role is: LEARNER
I20260620 23:11:36.742782 17880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:36.726906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38314 (local address 127.10.191.252:44225)
0620 23:11:36.727078 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:36.727085 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:36.727107 (+ 22us) server_negotiation.cc:407] Connection header received
0620 23:11:36.727233 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:36.727238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:36.727308 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:36.727410 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:36.728228 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.729632 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.731774 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.732561 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.732681 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:36.736848 (+ 4167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:36.736854 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:36.736859 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:36.736891 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:36.739923 (+ 3032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:36.740511 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:36.740516 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:36.740521 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:36.740630 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:36.740991 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:36.740996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:36.740999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:36.741383 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:36.741534 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:36.742031 (+ 497us) server_negotiation.cc:299] Negotiation successful
0620 23:11:36.742255 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 23:11:36.743494 17884 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d: attempting to promote NON_VOTER 6db99becd3324a5aaaf1eaf5985ae3f0 to VOTER
I20260620 23:11:36.744439 17884 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: false } }
I20260620 23:11:36.746327 17771 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [term 1 LEARNER]: Refusing update from remote peer c8e2e42e0532441a97821bf64376c25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 23:11:36.746858 17883 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 23:11:36.748812 17877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c: Bootstrap starting.
I20260620 23:11:36.749951 17884 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6db99becd3324a5aaaf1eaf5985ae3f0 (127.10.191.253) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: false } } }
I20260620 23:11:36.750612 17771 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6db99becd3324a5aaaf1eaf5985ae3f0 (127.10.191.253) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: false } } }
I20260620 23:11:36.753077 17877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:36.753764 17799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8e2e42e0532441a97821bf64376c25d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: false } } }
I20260620 23:11:36.753973 17799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:11:36.754312 17883 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c8e2e42e0532441a97821bf64376c25d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: false } } }
I20260620 23:11:36.754436 17877 log.cc:824] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c: Log is configured to *not* fsync() on all Append() calls
I20260620 23:11:36.754444 17885 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8e2e42e0532441a97821bf64376c25d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: false } } }
I20260620 23:11:36.754714 17883 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [sys.catalog]: This master's current role is: LEADER
I20260620 23:11:36.754882 17885 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [sys.catalog]: This master's current role is: LEADER
I20260620 23:11:36.757253 17877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c: No bootstrap required, opened a new log
I20260620 23:11:36.764676 17877 raft_consensus.cc:348] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } } peers { permanent_uuid: "806cc6e3c2fa46b7a43e12917c49150c" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 44225 } }
I20260620 23:11:36.765003 17877 raft_consensus.cc:374] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:36.765115 17877 raft_consensus.cc:729] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 806cc6e3c2fa46b7a43e12917c49150c, State: Initialized, Role: FOLLOWER
I20260620 23:11:36.765673 17877 consensus_queue.cc:260] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } } peers { permanent_uuid: "806cc6e3c2fa46b7a43e12917c49150c" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 44225 } }
I20260620 23:11:36.766466 17891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } } peers { permanent_uuid: "806cc6e3c2fa46b7a43e12917c49150c" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 44225 } } }
I20260620 23:11:36.766757 17891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:11:36.767549 17877 sys_catalog.cc:564] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [sys.catalog]: configured and running, proceeding with master startup.
W20260620 23:11:36.777454 17902 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:11:36.777592 17902 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:11:36.780095 17719 master_runner.cc:428] Detected that this master 806cc6e3c2fa46b7a43e12917c49150c is joining an existing cluster
I20260620 23:11:36.780192 17719 master_runner.cc:432] Initiating AddMaster RPC to add 127.10.191.252:44225
I20260620 23:11:36.801769 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:36.785314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42442 (local address 127.10.191.253:33657)
0620 23:11:36.785490 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:36.785498 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:36.785913 (+ 415us) server_negotiation.cc:407] Connection header received
0620 23:11:36.786235 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:36.786243 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:36.786322 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:36.786385 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:36.788010 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.789104 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.791278 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.791928 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.792022 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:36.794672 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:36.794689 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:36.794695 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:36.794737 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:36.798184 (+ 3447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:36.799299 (+ 1115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:36.799305 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:36.799308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:36.799486 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:36.799999 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:36.800004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:36.800008 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:36.800377 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:36.800513 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:36.801137 (+ 624us) server_negotiation.cc:299] Negotiation successful
0620 23:11:36.801369 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 23:11:36.802832 17909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:36.785230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:33657 (local address 127.0.0.1:42442)
0620 23:11:36.785805 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:36.785826 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:36.786061 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:36.786574 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:36.786579 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:36.787444 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:36.787813 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:36.787831 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.789304 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.789310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:36.791103 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:36.791116 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.792310 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.792320 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:36.792529 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:36.793253 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:36.793276 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:36.794463 (+ 1187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:36.798581 (+ 4118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:36.798586 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:36.798589 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:36.799138 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:36.799643 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:36.799648 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:36.799653 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:36.799853 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:36.800592 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:36.800599 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:36.800817 (+ 218us) client_negotiation.cc:769] Sending connection context
0620 23:11:36.801967 (+ 1150us) client_negotiation.cc:240] Negotiation successful
0620 23:11:36.802299 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":266,"threads_started":1}
I20260620 23:11:36.803179 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:36.784219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38230 (local address 127.10.191.254:39679)
0620 23:11:36.784426 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:36.784433 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:36.785065 (+ 632us) server_negotiation.cc:407] Connection header received
0620 23:11:36.785264 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:36.785270 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:36.785364 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:36.785467 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:36.788911 (+ 3444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.790364 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.793129 (+ 2765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.794079 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.794202 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:36.796949 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:36.796955 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:36.796958 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:36.796989 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:36.800061 (+ 3072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:36.800862 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:36.800867 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:36.800873 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:36.801022 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:36.801488 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:36.801493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:36.801497 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:36.801811 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:36.801940 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:36.802545 (+ 605us) server_negotiation.cc:299] Negotiation successful
0620 23:11:36.802790 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:11:36.804164 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:36.784286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.0.0.1:38230)
0620 23:11:36.784954 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:36.784973 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:36.785104 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:36.785690 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:36.785700 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:36.788221 (+ 2521us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:36.788683 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:36.788700 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.790622 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.790630 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:36.792934 (+ 2304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:36.792955 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.794587 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.794599 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:36.794896 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:36.795842 (+ 946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:36.795857 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:36.796736 (+ 879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:36.800208 (+ 3472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:36.800229 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:36.800233 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:36.800720 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:36.801160 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:36.801164 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:36.801167 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:36.801363 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:36.802015 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:36.802019 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:36.802262 (+ 243us) client_negotiation.cc:769] Sending connection context
0620 23:11:36.803623 (+ 1361us) client_negotiation.cc:240] Negotiation successful
0620 23:11:36.803886 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":347,"threads_started":1}
I20260620 23:11:36.814286 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 17219
I20260620 23:11:36.814483 17880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:36.786047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38330 (local address 127.10.191.252:44225)
0620 23:11:36.786299 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:36.786305 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:36.798433 (+ 12128us) server_negotiation.cc:407] Connection header received
0620 23:11:36.798619 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:36.798624 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:36.798734 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:36.798811 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:36.800157 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.801312 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.804390 (+ 3078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.805111 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.805202 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:36.807686 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:36.807692 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:36.807696 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:36.807775 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:36.811366 (+ 3591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:36.812186 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:36.812192 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:36.812195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:36.812370 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:36.812799 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:36.812804 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:36.812808 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:36.813145 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:36.813290 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:36.813838 (+ 548us) server_negotiation.cc:299] Negotiation successful
0620 23:11:36.814134 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 23:11:36.815505 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:36.786506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:44225 (local address 127.0.0.1:38330)
0620 23:11:36.798356 (+ 11850us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:36.798375 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:36.798510 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:36.799012 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:36.799017 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:36.799643 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:36.800012 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:36.800023 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.801518 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.801524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:36.803600 (+ 2076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:36.804254 (+ 654us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.805484 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.805490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:36.805644 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:36.806279 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:36.806321 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:36.807496 (+ 1175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:36.811507 (+ 4011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:36.811515 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:36.811519 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:36.812049 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:36.812504 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:36.812509 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:36.812512 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:36.812651 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:36.813360 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:36.813364 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:36.813517 (+ 153us) client_negotiation.cc:769] Sending connection context
0620 23:11:36.814692 (+ 1175us) client_negotiation.cc:240] Negotiation successful
0620 23:11:36.815000 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10014,"thread_start_us":9012,"threads_started":1}
Jun 20 23:11:36 dist-test-slave-jkbq krb5kdc[17113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997093, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.191.254@KRBTEST.COM for oryx/127.10.191.252@KRBTEST.COM
I20260620 23:11:36.825333 17886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:36.809304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:44225 (local address 127.0.0.1:38342)
0620 23:11:36.809721 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:36.809740 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:36.809848 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:36.810452 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:36.810458 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:36.810974 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:36.811155 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:36.811167 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.812638 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.812644 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:36.813923 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:36.813938 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.815025 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.815030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:36.815230 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:36.815821 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:36.815843 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:36.818794 (+ 2951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:36.822706 (+ 3912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:36.822712 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:36.822716 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:36.823273 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:36.823687 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:36.823692 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:36.823696 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:36.823837 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:36.824358 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:36.824364 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:36.824566 (+ 202us) client_negotiation.cc:769] Sending connection context
0620 23:11:36.824705 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 23:11:36.825003 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290}
I20260620 23:11:36.826447 17911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:36.809420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38342 (local address 127.10.191.252:44225)
0620 23:11:36.810069 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:36.810080 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:36.810105 (+ 25us) server_negotiation.cc:407] Connection header received
0620 23:11:36.810189 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:36.810195 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:36.810284 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:36.810403 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:36.811333 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.812524 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.814086 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.814806 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.818997 (+ 4191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:36.819257 (+ 260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:36.819266 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:36.819273 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:36.819317 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:36.822537 (+ 3220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:36.823412 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:36.823416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:36.823419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:36.823592 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:36.823956 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:36.823961 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:36.823964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:36.824235 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:36.824384 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:36.825795 (+ 1411us) server_negotiation.cc:299] Negotiation successful
0620 23:11:36.826010 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":561,"threads_started":1}
Jun 20 23:11:36 dist-test-slave-jkbq krb5kdc[17113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.191.193@KRBTEST.COM
I20260620 23:11:36.835362 17912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:36.816257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53384 (local address 127.10.191.193:40211)
0620 23:11:36.818100 (+ 1843us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:36.818109 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:36.818138 (+ 29us) server_negotiation.cc:407] Connection header received
0620 23:11:36.818207 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:36.818216 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:36.818516 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:36.818698 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:36.820397 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.821987 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.823596 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.824481 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.824610 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:36.829643 (+ 5033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:36.829655 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:36.829669 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:36.829707 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:36.832509 (+ 2802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:36.833189 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:36.833195 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:36.833198 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:36.833321 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:36.833724 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:36.833729 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:36.833734 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:36.834057 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:36.834367 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:36.834685 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 23:11:36.834971 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1673,"thread_start_us":1516,"threads_started":1}
I20260620 23:11:36.836619 17342 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:11:36.838719 17158 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.10.191.252:44225: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "806cc6e3c2fa46b7a43e12917c49150c"
member_type: NON_VOTER
last_known_addr {
host: "127.10.191.252"
port: 44225
}
attrs {
promote: true
}
}
dest_uuid: "c8e2e42e0532441a97821bf64376c25d"
cas_config_opid_index: 6
I20260620 23:11:36.839772 17158 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: false } } peers { permanent_uuid: "806cc6e3c2fa46b7a43e12917c49150c" member_type: NON_VOTER last_known_addr { host: "127.10.191.252" port: 44225 } attrs { promote: true } }
I20260620 23:11:36.843314 17852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:36.848286 17852 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [term 1 FOLLOWER]: Refusing update from remote peer c8e2e42e0532441a97821bf64376c25d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 23:11:36.849651 17771 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [term 1 FOLLOWER]: Refusing update from remote peer c8e2e42e0532441a97821bf64376c25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 23:11:36.850440 17889 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 23:11:36.851194 17889 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "806cc6e3c2fa46b7a43e12917c49150c" member_type: NON_VOTER last_known_addr { host: "127.10.191.252" port: 44225 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:11:36.853703 17885 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 806cc6e3c2fa46b7a43e12917c49150c (127.10.191.252) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: false } } peers { permanent_uuid: "806cc6e3c2fa46b7a43e12917c49150c" member_type: NON_VOTER last_known_addr { host: "127.10.191.252" port: 44225 } attrs { promote: true } } }
I20260620 23:11:36.854264 17771 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 806cc6e3c2fa46b7a43e12917c49150c (127.10.191.252) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: false } } peers { permanent_uuid: "806cc6e3c2fa46b7a43e12917c49150c" member_type: NON_VOTER last_known_addr { host: "127.10.191.252" port: 44225 } attrs { promote: true } } }
I20260620 23:11:36.857437 17885 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.10.191.252:44225
I20260620 23:11:36.857893 17884 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c8e2e42e0532441a97821bf64376c25d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: false } } peers { permanent_uuid: "806cc6e3c2fa46b7a43e12917c49150c" member_type: NON_VOTER last_known_addr { host: "127.10.191.252" port: 44225 } attrs { promote: true } } }
I20260620 23:11:36.858166 17719 master.cc:537] Master@127.10.191.252:44225 shutting down...
I20260620 23:11:36.858182 17884 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [sys.catalog]: This master's current role is: LEADER
I20260620 23:11:36.858507 17884 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8e2e42e0532441a97821bf64376c25d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: false } } peers { permanent_uuid: "806cc6e3c2fa46b7a43e12917c49150c" member_type: NON_VOTER last_known_addr { host: "127.10.191.252" port: 44225 } attrs { promote: true } } }
I20260620 23:11:36.858752 17884 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [sys.catalog]: This master's current role is: LEADER
I20260620 23:11:36.861832 17799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8e2e42e0532441a97821bf64376c25d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: false } } peers { permanent_uuid: "806cc6e3c2fa46b7a43e12917c49150c" member_type: NON_VOTER last_known_addr { host: "127.10.191.252" port: 44225 } attrs { promote: true } } }
I20260620 23:11:36.862142 17799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:11:36.875954 17852 raft_consensus.cc:858] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } } peers { permanent_uuid: "806cc6e3c2fa46b7a43e12917c49150c" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 44225 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: NON_VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: false } }
W20260620 23:11:36.876116 17852 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } } peers { permanent_uuid: "806cc6e3c2fa46b7a43e12917c49150c" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 44225 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: NON_VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: false } }
I20260620 23:11:36.883865 17852 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 6db99becd3324a5aaaf1eaf5985ae3f0 (127.10.191.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: NON_VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: true } } }
I20260620 23:11:36.894351 17891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8e2e42e0532441a97821bf64376c25d. Latest consensus state: current_term: 1 leader_uuid: "c8e2e42e0532441a97821bf64376c25d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: NON_VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: true } } }
I20260620 23:11:36.894636 17891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 23:11:36.894485 17884 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8e2e42e0532441a97821bf64376c25d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: false } } peers { permanent_uuid: "806cc6e3c2fa46b7a43e12917c49150c" member_type: NON_VOTER last_known_addr { host: "127.10.191.252" port: 44225 } attrs { promote: true } } }
I20260620 23:11:36.894794 17884 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [sys.catalog]: This master's current role is: LEADER
I20260620 23:11:36.895267 17916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8e2e42e0532441a97821bf64376c25d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: NON_VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: true } } }
I20260620 23:11:36.895571 17916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 23:11:36.895750 17920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8e2e42e0532441a97821bf64376c25d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: NON_VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: true } } }
I20260620 23:11:36.896050 17920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [sys.catalog]: This master's current role is: NON_PARTICIPANT
W20260620 23:11:36.896030 17145 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260620 23:11:36.897593 17145 consensus_peers.cc:487] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d -> Peer 806cc6e3c2fa46b7a43e12917c49150c (127.10.191.252:44225): Couldn't send request to peer 806cc6e3c2fa46b7a43e12917c49150c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 23:11:36.899283 17719 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260620 23:11:36.899747 17719 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260620 23:11:36.899926 17719 tablet_replica.cc:330] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c: stopping tablet replica
I20260620 23:11:36.908288 17719 master.cc:552] Master@127.10.191.252:44225 shutdown complete.
I20260620 23:11:36.908422 17719 master_runner.cc:305] Clearing existing system tablet
I20260620 23:11:36.908841 17719 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:36.909006 17719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:36.912211 17719 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 23:11:36.913935 17923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:36.914338 17719 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:36.914731 17719 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 23:11:36.914891 17719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-2/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-2/wal
uuid: "806cc6e3c2fa46b7a43e12917c49150c"
format_stamp: "Formatted at 2026-06-20 23:11:36 on dist-test-slave-jkbq"
server_key: "0e718311e3f81fb2aad8f68c994b427a"
server_key_iv: "bc9a14ea3037721246ab9bdb830153dd"
server_key_version: "encryptionkey@0"
I20260620 23:11:36.919584 17719 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:11:36.926465 17719 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 23:11:36.926692 17719 log.cc:1196] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 23:11:36.927268 17719 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c: Deleting consensus metadata
I20260620 23:11:36.927546 17719 master_runner.cc:315] Copying system tablet from 127.10.191.254:39679
I20260620 23:11:36.930006 17719 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c: tablet copy: Beginning tablet copy session from remote peer at address 127.10.191.254:39679
I20260620 23:11:36.949211 17929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:36.930830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.0.0.1:38232)
0620 23:11:36.935316 (+ 4486us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:36.935337 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:36.935569 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:36.935994 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:36.935999 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:36.936514 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:36.936780 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:36.936793 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.938293 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.938299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:36.939918 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:36.939933 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.940904 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.940911 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:36.941096 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:36.941827 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:36.941850 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:36.942922 (+ 1072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:36.946490 (+ 3568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:36.946496 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:36.946501 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:36.947001 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:36.947400 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:36.947405 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:36.947409 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:36.947579 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:36.948150 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:36.948160 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:36.948333 (+ 173us) client_negotiation.cc:769] Sending connection context
0620 23:11:36.948470 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 23:11:36.948765 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4345,"thread_start_us":4149,"threads_started":1}
I20260620 23:11:36.949239 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:36.930944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38232 (local address 127.10.191.254:39679)
0620 23:11:36.931181 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:36.931188 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:36.935442 (+ 4254us) server_negotiation.cc:407] Connection header received
0620 23:11:36.935739 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:36.935745 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:36.935834 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:36.935949 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:36.936940 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.938144 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.940067 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:36.940728 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:36.940857 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:36.943140 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:36.943146 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:36.943151 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:36.943188 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:36.946357 (+ 3169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:36.947134 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:36.947139 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:36.947143 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:36.947289 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:36.947707 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:36.947712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:36.947715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:36.948020 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:36.948180 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:36.948607 (+ 427us) server_negotiation.cc:299] Negotiation successful
0620 23:11:36.948816 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260620 23:11:36.949961 17178 tablet_copy_service.cc:140] P c8e2e42e0532441a97821bf64376c25d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 806cc6e3c2fa46b7a43e12917c49150c ({username='oryx', principal='oryx/127.10.191.252@KRBTEST.COM'} at 127.0.0.1:38232)
I20260620 23:11:36.950150 17178 tablet_copy_service.cc:161] P c8e2e42e0532441a97821bf64376c25d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 806cc6e3c2fa46b7a43e12917c49150c at {username='oryx', principal='oryx/127.10.191.252@KRBTEST.COM'} at 127.0.0.1:38232: session id = 806cc6e3c2fa46b7a43e12917c49150c-00000000000000000000000000000000
I20260620 23:11:36.953670 17178 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 23:11:36.955528 17719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 23:11:36.962265 17719 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c: tablet copy: Starting download of 0 data blocks...
I20260620 23:11:36.962570 17719 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c: tablet copy: Starting download of 1 WAL segments...
I20260620 23:11:36.965013 17719 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 23:11:36.969466 17178 tablet_copy_service.cc:342] P c8e2e42e0532441a97821bf64376c25d: Request end of tablet copy session 806cc6e3c2fa46b7a43e12917c49150c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.10.191.252@KRBTEST.COM'} at 127.0.0.1:38232
I20260620 23:11:36.969604 17178 tablet_copy_service.cc:433] P c8e2e42e0532441a97821bf64376c25d: ending tablet copy session 806cc6e3c2fa46b7a43e12917c49150c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 806cc6e3c2fa46b7a43e12917c49150c
I20260620 23:11:36.981060 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 17219
I20260620 23:11:36.981408 17719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:36.984105 17930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:36.988291 17931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:36.999004 17719 server_base.cc:1034] running on GCE node
W20260620 23:11:36.998901 17933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:36.999635 17719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:37.000289 17719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:37.001494 17719 hybrid_clock.cc:648] HybridClock initialized: now 1781997097001436 us; error 51 us; skew 500 ppm
I20260620 23:11:37.001750 17719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:11:36 dist-test-slave-jkbq krb5kdc[17113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997096, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.191.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:37.003681 17719 init.cc:377] Logged in from keytab as oryx/127.10.191.252@KRBTEST.COM (short username oryx)
I20260620 23:11:37.008935 17719 webserver.cc:460] Webserver started at http://127.10.191.252:38033/ using document root <none> and password file <none>
I20260620 23:11:37.009290 17719 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:37.009263 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.193:40211
--local_ip_for_outbound_sockets=127.10.191.193
--tserver_master_addrs=127.10.191.254:39679
--webserver_port=37087
--webserver_interface=127.10.191.193
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.193.keytab
--principal=oryx/127.10.191.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:33741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.191.254:39679,127.10.191.253:33657
--tserver_master_addrs=127.10.191.254:39679,127.10.191.253:33657,127.10.191.252:44225 with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:11:37.009400 17719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:37.012746 17719 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 23:11:37.014793 17940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:37.015347 17719 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:11:37.015479 17719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-2/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-2/wal
uuid: "806cc6e3c2fa46b7a43e12917c49150c"
format_stamp: "Formatted at 2026-06-20 23:11:36 on dist-test-slave-jkbq"
server_key: "0e718311e3f81fb2aad8f68c994b427a"
server_key_iv: "bc9a14ea3037721246ab9bdb830153dd"
server_key_version: "encryptionkey@0"
I20260620 23:11:37.015678 17719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:37.027254 17719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:37.030728 17719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:37.057363 17719 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.252:44225
I20260620 23:11:37.057385 17992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.252:44225 every 8 connection(s)
I20260620 23:11:37.058252 17719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/master-2/data/info.pb
I20260620 23:11:37.065502 17993 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:11:37.070320 17993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c: Bootstrap starting.
I20260620 23:11:37.085995 17993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:11:37.086468 17993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c: Bootstrap complete.
I20260620 23:11:37.087383 17993 raft_consensus.cc:348] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: false } } peers { permanent_uuid: "806cc6e3c2fa46b7a43e12917c49150c" member_type: NON_VOTER last_known_addr { host: "127.10.191.252" port: 44225 } attrs { promote: true } }
I20260620 23:11:37.087651 17993 raft_consensus.cc:729] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 806cc6e3c2fa46b7a43e12917c49150c, State: Initialized, Role: LEARNER
I20260620 23:11:37.087867 17993 consensus_queue.cc:260] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: false } } peers { permanent_uuid: "806cc6e3c2fa46b7a43e12917c49150c" member_type: NON_VOTER last_known_addr { host: "127.10.191.252" port: 44225 } attrs { promote: true } }
I20260620 23:11:37.089593 17996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: false } } peers { permanent_uuid: "806cc6e3c2fa46b7a43e12917c49150c" member_type: NON_VOTER last_known_addr { host: "127.10.191.252" port: 44225 } attrs { promote: true } } }
I20260620 23:11:37.089921 17993 sys_catalog.cc:564] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:11:37.090031 17996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [sys.catalog]: This master's current role is: LEARNER
I20260620 23:11:37.110482 18007 catalog_manager.cc:1259] Loaded cluster ID: c21dc8b8f31a4eff9c5507da7807c540
I20260620 23:11:37.110605 18007 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c: loading cluster ID for follower catalog manager: success
I20260620 23:11:37.113263 18007 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c: acquiring CA information for follower catalog manager: success
I20260620 23:11:37.115056 18007 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:11:37.328979 17884 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8e2e42e0532441a97821bf64376c25d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: false } } peers { permanent_uuid: "806cc6e3c2fa46b7a43e12917c49150c" member_type: NON_VOTER last_known_addr { host: "127.10.191.252" port: 44225 } attrs { promote: true } } }
I20260620 23:11:37.329321 17884 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [sys.catalog]: This master's current role is: LEADER
I20260620 23:11:37.346375 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:37.329386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:44225 (local address 127.0.0.1:38354)
0620 23:11:37.340020 (+ 10634us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:37.340038 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:37.340181 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:37.340593 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:37.340597 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:37.340617 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:11:37.340781 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:37.340789 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:37.342766 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:37.342771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:37.344100 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:37.344114 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:37.345821 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:37.345826 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:37.345940 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:37.345974 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 23:11:37.346043 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 23:11:37.346140 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10508,"thread_start_us":10410,"threads_started":1}
I20260620 23:11:37.346607 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:37.329791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38354 (local address 127.10.191.252:44225)
0620 23:11:37.339867 (+ 10076us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:37.339875 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:37.340161 (+ 286us) server_negotiation.cc:407] Connection header received
0620 23:11:37.340317 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:37.340325 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:37.340429 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:37.340577 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:11:37.340916 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:37.342543 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:37.344265 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:37.345640 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:37.345827 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:37.346028 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:37.346084 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 23:11:37.346236 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9902,"thread_start_us":9722,"threads_started":1}
I20260620 23:11:37.347405 17968 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.7] Dedup: 1.7->[]
I20260620 23:11:37.347903 17996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8e2e42e0532441a97821bf64376c25d. Latest consensus state: current_term: 1 leader_uuid: "c8e2e42e0532441a97821bf64376c25d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: false } } peers { permanent_uuid: "806cc6e3c2fa46b7a43e12917c49150c" member_type: NON_VOTER last_known_addr { host: "127.10.191.252" port: 44225 } attrs { promote: true } } }
I20260620 23:11:37.348210 17996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [sys.catalog]: This master's current role is: LEARNER
I20260620 23:11:37.348572 17889 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d: attempting to promote NON_VOTER 806cc6e3c2fa46b7a43e12917c49150c to VOTER
I20260620 23:11:37.349146 17889 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: false } } peers { permanent_uuid: "806cc6e3c2fa46b7a43e12917c49150c" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 44225 } attrs { promote: false } }
I20260620 23:11:37.350950 17968 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [term 1 LEARNER]: Refusing update from remote peer c8e2e42e0532441a97821bf64376c25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 23:11:37.351034 17771 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [term 1 FOLLOWER]: Refusing update from remote peer c8e2e42e0532441a97821bf64376c25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 23:11:37.351568 17885 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 23:11:37.351867 17884 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "806cc6e3c2fa46b7a43e12917c49150c" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 44225 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 23:11:37.354614 17885 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 806cc6e3c2fa46b7a43e12917c49150c (127.10.191.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: false } } peers { permanent_uuid: "806cc6e3c2fa46b7a43e12917c49150c" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 44225 } attrs { promote: false } } }
I20260620 23:11:37.358404 17889 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c8e2e42e0532441a97821bf64376c25d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: false } } peers { permanent_uuid: "806cc6e3c2fa46b7a43e12917c49150c" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 44225 } attrs { promote: false } } }
I20260620 23:11:37.358697 17889 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [sys.catalog]: This master's current role is: LEADER
I20260620 23:11:37.359104 17889 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8e2e42e0532441a97821bf64376c25d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: false } } peers { permanent_uuid: "806cc6e3c2fa46b7a43e12917c49150c" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 44225 } attrs { promote: false } } }
I20260620 23:11:37.359336 17889 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [sys.catalog]: This master's current role is: LEADER
I20260620 23:11:37.359676 17771 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 806cc6e3c2fa46b7a43e12917c49150c (127.10.191.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: false } } peers { permanent_uuid: "806cc6e3c2fa46b7a43e12917c49150c" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 44225 } attrs { promote: false } } }
I20260620 23:11:37.360162 17885 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8e2e42e0532441a97821bf64376c25d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: false } } peers { permanent_uuid: "806cc6e3c2fa46b7a43e12917c49150c" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 44225 } attrs { promote: false } } }
I20260620 23:11:37.360432 17885 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d [sys.catalog]: This master's current role is: LEADER
I20260620 23:11:37.360760 17968 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 806cc6e3c2fa46b7a43e12917c49150c (127.10.191.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: false } } peers { permanent_uuid: "806cc6e3c2fa46b7a43e12917c49150c" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 44225 } attrs { promote: false } } }
I20260620 23:11:37.364022 17996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8e2e42e0532441a97821bf64376c25d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: false } } peers { permanent_uuid: "806cc6e3c2fa46b7a43e12917c49150c" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 44225 } attrs { promote: false } } }
I20260620 23:11:37.364259 17996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:11:37.364302 18011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8e2e42e0532441a97821bf64376c25d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: false } } peers { permanent_uuid: "806cc6e3c2fa46b7a43e12917c49150c" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 44225 } attrs { promote: false } } }
I20260620 23:11:37.364621 18011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 23:11:37.406921 17941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:37.407186 17941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:37.407279 17941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:37.414428 17941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:37.414525 17941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:37.414641 17941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.193
I20260620 23:11:37.424258 17941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:33741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.193.keytab
--principal=oryx/127.10.191.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.193:40211
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.191.193
--webserver_port=37087
--tserver_master_addrs=127.10.191.254:39679,127.10.191.253:33657,127.10.191.252:44225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.193
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:37.425448 17941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:37.426658 17941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:37.439347 18019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:37.439319 18016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:37.439462 18017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:37.440424 17941 server_base.cc:1034] running on GCE node
I20260620 23:11:37.441031 17941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:37.441960 17941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:37.443185 17941 hybrid_clock.cc:648] HybridClock initialized: now 1781997097443136 us; error 61 us; skew 500 ppm
I20260620 23:11:37.443518 17941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:11:37 dist-test-slave-jkbq krb5kdc[17113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997097, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.191.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:37.446635 17941 init.cc:377] Logged in from keytab as oryx/127.10.191.193@KRBTEST.COM (short username oryx)
I20260620 23:11:37.448176 17941 webserver.cc:460] Webserver started at http://127.10.191.193:37087/ using document root <none> and password file <none>
I20260620 23:11:37.448897 17941 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:37.449054 17941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:37.454300 17941 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:11:37.457031 18026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:37.458271 17941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 23:11:37.458446 17941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-0/wal
uuid: "f3868efeb1fa4926a2fa269532dddbab"
format_stamp: "Formatted at 2026-06-20 23:11:34 on dist-test-slave-jkbq"
server_key: "947167ef544ab69f8889d24103eb920a"
server_key_iv: "229517d80a07e286b00120b7a29e9f4a"
server_key_version: "encryptionkey@0"
I20260620 23:11:37.459549 17941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:37.481348 17941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:37.485340 17941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:37.485620 17941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:37.486541 17941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:11:37.487905 17941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:11:37.488030 17941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:37.488157 17941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:11:37.488246 17941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:37.522032 17941 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.193:40211
I20260620 23:11:37.522119 18138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.193:40211 every 8 connection(s)
I20260620 23:11:37.523523 17941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
I20260620 23:11:37.527257 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 17941
I20260620 23:11:37.527626 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 17353
Jun 20 23:11:37 dist-test-slave-jkbq krb5kdc[17113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997097, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.191.193@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM
I20260620 23:11:37.553453 18143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:37.526559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:40955 (local address 127.10.191.254:39679)
0620 23:11:37.527457 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:37.527464 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:37.528321 (+ 857us) server_negotiation.cc:407] Connection header received
0620 23:11:37.531797 (+ 3476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:37.531806 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:37.531919 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:37.532019 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:37.534850 (+ 2831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:37.536501 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:37.539786 (+ 3285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:37.540698 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:37.540812 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:37.545568 (+ 4756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:37.545576 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:37.545582 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:37.545626 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:37.549284 (+ 3658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:37.550329 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:37.550334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:37.550338 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:37.550482 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:37.551045 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:37.551049 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:37.551052 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:37.551403 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:37.551536 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:37.552590 (+ 1054us) server_negotiation.cc:299] Negotiation successful
0620 23:11:37.552883 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":760,"thread_start_us":647,"threads_started":1}
Jun 20 23:11:37 dist-test-slave-jkbq krb5kdc[17113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997097, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.191.193@KRBTEST.COM for oryx/127.10.191.253@KRBTEST.COM
I20260620 23:11:37.556478 18144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:37.527297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.10.191.193:40955)
0620 23:11:37.528145 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:37.528221 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:37.531382 (+ 3161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:37.532721 (+ 1339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:37.532738 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:37.533546 (+ 808us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:37.534563 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:37.534597 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:37.536806 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:37.536815 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:37.539550 (+ 2735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:37.539570 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:37.541203 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:37.541213 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:37.541530 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:37.542306 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:37.542334 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:37.545279 (+ 2945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:37.549494 (+ 4215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:37.549504 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:37.549525 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:37.550196 (+ 671us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:37.550619 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:37.550624 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:37.550628 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:37.550905 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:37.551630 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:37.551638 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:37.552151 (+ 513us) client_negotiation.cc:769] Sending connection context
0620 23:11:37.553992 (+ 1841us) client_negotiation.cc:240] Negotiation successful
0620 23:11:37.554645 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":336,"threads_started":1}
Jun 20 23:11:37 dist-test-slave-jkbq krb5kdc[17113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.191.194@KRBTEST.COM
I20260620 23:11:37.560286 18140 heartbeater.cc:347] Connected to a master server at 127.10.191.254:39679
I20260620 23:11:37.560679 18140 heartbeater.cc:464] Registering TS with master...
Jun 20 23:11:37 dist-test-slave-jkbq krb5kdc[17113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997097, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.191.193@KRBTEST.COM for oryx/127.10.191.252@KRBTEST.COM
I20260620 23:11:37.561698 18140 heartbeater.cc:511] Master 127.10.191.254:39679 requested a full tablet report, sending...
I20260620 23:11:37.564007 18146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:37.528674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:33657 (local address 127.10.191.193:52607)
0620 23:11:37.544951 (+ 16277us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:37.544971 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:37.545102 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:37.546096 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:37.546103 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:37.546932 (+ 829us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:37.547343 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:37.547358 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:37.549284 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:37.549293 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:37.551457 (+ 2164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:37.551471 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:37.552643 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:37.552651 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:37.552876 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:37.553406 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:37.553433 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:37.556363 (+ 2930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:37.560554 (+ 4191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:37.560561 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:37.560566 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:37.561200 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:37.561641 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:37.561646 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:37.561649 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:37.561802 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:37.562918 (+ 1116us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:37.562923 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:37.563072 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 23:11:37.563204 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 23:11:37.563562 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16131,"thread_start_us":15927,"threads_started":1}
I20260620 23:11:37.563987 17158 ts_manager.cc:194] Re-registered known tserver with Master: f3868efeb1fa4926a2fa269532dddbab (127.10.191.193:40211)
I20260620 23:11:37.565188 18147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:37.528798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:52607 (local address 127.10.191.253:33657)
0620 23:11:37.544147 (+ 15349us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:37.544156 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:37.545451 (+ 1295us) server_negotiation.cc:407] Connection header received
0620 23:11:37.545544 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:37.545551 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:37.545682 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:37.545820 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:37.547565 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:37.549072 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:37.551646 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:37.552267 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:37.552382 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:37.556757 (+ 4375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:37.556767 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:37.556771 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:37.556813 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:37.560390 (+ 3577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:37.561348 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:37.561355 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:37.561359 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:37.561494 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:37.562281 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:37.562288 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:37.562294 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:37.562692 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:37.562844 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:37.564562 (+ 1718us) server_negotiation.cc:299] Negotiation successful
0620 23:11:37.564800 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15152,"thread_start_us":15373,"threads_started":1}
I20260620 23:11:37.565176 17158 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.191.193@KRBTEST.COM'} at 127.10.191.193:40955
I20260620 23:11:37.566397 18148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:37.529294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56496 (local address 127.10.191.194:35513)
0620 23:11:37.543738 (+ 14444us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:37.543747 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:37.543776 (+ 29us) server_negotiation.cc:407] Connection header received
0620 23:11:37.543848 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:37.543856 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:37.544153 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:37.544313 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:37.547399 (+ 3086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:37.549115 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:37.551054 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:37.551963 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:37.552055 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:37.558231 (+ 6176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:37.558246 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:37.558262 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:37.558309 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:37.562119 (+ 3810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:37.563446 (+ 1327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:37.563454 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:37.563456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:37.563616 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:37.564453 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:37.564458 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:37.564462 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:37.564859 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:37.565119 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:37.565701 (+ 582us) server_negotiation.cc:299] Negotiation successful
0620 23:11:37.565986 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14249,"thread_start_us":12738,"threads_started":1}
I20260620 23:11:37.566854 18141 heartbeater.cc:347] Connected to a master server at 127.10.191.253:33657
I20260620 23:11:37.567018 18141 heartbeater.cc:464] Registering TS with master...
I20260620 23:11:37.567555 18141 heartbeater.cc:511] Master 127.10.191.253:33657 requested a full tablet report, sending...
I20260620 23:11:37.568958 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:37.545170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:55977 (local address 127.10.191.252:44225)
0620 23:11:37.545390 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:37.545398 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:37.551762 (+ 6364us) server_negotiation.cc:407] Connection header received
0620 23:11:37.552234 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:37.552240 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:37.552324 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:37.552390 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:37.553746 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:37.554845 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:37.556787 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:37.557420 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:37.557516 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:37.562177 (+ 4661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:37.562183 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:37.562187 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:37.562229 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:37.565799 (+ 3570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:37.566784 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:37.566789 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:37.566792 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:37.566963 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:37.567430 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:37.567435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:37.567438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:37.567770 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:37.567896 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:37.568384 (+ 488us) server_negotiation.cc:299] Negotiation successful
0620 23:11:37.568632 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 23:11:37.569047 17476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:11:37.569516 17761 ts_manager.cc:194] Registered new tserver with Master: f3868efeb1fa4926a2fa269532dddbab (127.10.191.193:40211)
I20260620 23:11:37.570259 18149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:37.545049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:44225 (local address 127.10.191.193:55977)
0620 23:11:37.551665 (+ 6616us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:37.551683 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:37.552063 (+ 380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:37.552582 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:37.552589 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:37.553188 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:37.553557 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:37.553572 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:37.555041 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:37.555046 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:37.556648 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:37.556658 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:37.558222 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:37.558228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:37.558670 (+ 442us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:37.559255 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:37.559280 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:37.561941 (+ 2661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:37.565928 (+ 3987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:37.565936 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:37.565939 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:37.566391 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:37.567112 (+ 721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:37.567117 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:37.567121 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:37.567296 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:37.567990 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:37.567995 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:37.568136 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 23:11:37.569365 (+ 1229us) client_negotiation.cc:240] Negotiation successful
0620 23:11:37.569725 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6492,"thread_start_us":6725,"threads_started":1}
I20260620 23:11:37.571292 18139 heartbeater.cc:347] Connected to a master server at 127.10.191.252:44225
I20260620 23:11:37.571429 18139 heartbeater.cc:464] Registering TS with master...
I20260620 23:11:37.571830 18139 heartbeater.cc:511] Master 127.10.191.252:44225 requested a full tablet report, sending...
I20260620 23:11:37.573655 17958 ts_manager.cc:194] Registered new tserver with Master: f3868efeb1fa4926a2fa269532dddbab (127.10.191.193:40211)
I20260620 23:11:37.651576 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 17353
I20260620 23:11:37.668537 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.194:35513
--local_ip_for_outbound_sockets=127.10.191.194
--tserver_master_addrs=127.10.191.254:39679
--webserver_port=43957
--webserver_interface=127.10.191.194
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.194.keytab
--principal=oryx/127.10.191.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:33741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.191.254:39679,127.10.191.253:33657
--tserver_master_addrs=127.10.191.254:39679,127.10.191.253:33657,127.10.191.252:44225 with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:11:38.001076 18151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:38.001405 18151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:38.001575 18151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:38.008921 18151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:38.009083 18151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:38.009261 18151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.194
I20260620 23:11:38.019161 18151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:33741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.194.keytab
--principal=oryx/127.10.191.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.194:35513
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.191.194
--webserver_port=43957
--tserver_master_addrs=127.10.191.254:39679,127.10.191.253:33657,127.10.191.252:44225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.194
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:38.020493 18151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:38.021809 18151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:38.035226 18160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:38.035308 18157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:38.035284 18158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:38.035917 18151 server_base.cc:1034] running on GCE node
I20260620 23:11:38.036506 18151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:38.037416 18151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:38.038607 18151 hybrid_clock.cc:648] HybridClock initialized: now 1781997098038564 us; error 47 us; skew 500 ppm
I20260620 23:11:38.038991 18151 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:11:38 dist-test-slave-jkbq krb5kdc[17113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997098, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.191.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:38.042138 18151 init.cc:377] Logged in from keytab as oryx/127.10.191.194@KRBTEST.COM (short username oryx)
I20260620 23:11:38.043653 18151 webserver.cc:460] Webserver started at http://127.10.191.194:43957/ using document root <none> and password file <none>
I20260620 23:11:38.044320 18151 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:38.044466 18151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:38.049857 18151 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 23:11:38.052711 18167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:38.054078 18151 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:11:38.054248 18151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-1/wal
uuid: "ab48ee8fa5634fc59ab80e8a1c3d5361"
format_stamp: "Formatted at 2026-06-20 23:11:34 on dist-test-slave-jkbq"
server_key: "fb948fb8a31e1cc39ac43e97335f4287"
server_key_iv: "45011df2e26d736ac985ae8177ea6d84"
server_key_version: "encryptionkey@0"
I20260620 23:11:38.055269 18151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:38.073899 18151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:38.078023 18151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:38.078265 18151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:38.079188 18151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:11:38.080438 18151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:11:38.080520 18151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:38.080598 18151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:11:38.080663 18151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:38.113860 18151 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.194:35513
I20260620 23:11:38.113950 18279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.194:35513 every 8 connection(s)
I20260620 23:11:38.115468 18151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
I20260620 23:11:38.118448 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 18151
I20260620 23:11:38.118743 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 17486
Jun 20 23:11:38 dist-test-slave-jkbq krb5kdc[17113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997098, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.191.194@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM
Jun 20 23:11:38 dist-test-slave-jkbq krb5kdc[17113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997098, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.191.194@KRBTEST.COM for oryx/127.10.191.252@KRBTEST.COM
Jun 20 23:11:38 dist-test-slave-jkbq krb5kdc[17113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997098, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.191.194@KRBTEST.COM for oryx/127.10.191.253@KRBTEST.COM
Jun 20 23:11:38 dist-test-slave-jkbq krb5kdc[17113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.191.195@KRBTEST.COM
I20260620 23:11:38.148212 18289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:38.120969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:55421 (local address 127.10.191.252:44225)
0620 23:11:38.123631 (+ 2662us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:38.123640 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:38.124309 (+ 669us) server_negotiation.cc:407] Connection header received
0620 23:11:38.124379 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:38.124384 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:38.124473 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:38.124596 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:38.129619 (+ 5023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.131106 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.135347 (+ 4241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.136340 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.136446 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:38.140855 (+ 4409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:38.140863 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:38.140868 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:38.140914 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:38.144427 (+ 3513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:38.145411 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:38.145416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:38.145421 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:38.145561 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:38.146112 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:38.146117 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:38.146120 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:38.146463 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:38.146590 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:38.147591 (+ 1001us) server_negotiation.cc:299] Negotiation successful
0620 23:11:38.147843 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2509,"thread_start_us":2412,"threads_started":1}
I20260620 23:11:38.149670 18284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:38.118559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:45007 (local address 127.10.191.254:39679)
0620 23:11:38.125317 (+ 6758us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:38.125326 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:38.125355 (+ 29us) server_negotiation.cc:407] Connection header received
0620 23:11:38.125437 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:38.125445 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:38.125546 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:38.125707 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:38.128658 (+ 2951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.130153 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.134232 (+ 4079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.134646 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.134790 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:38.141391 (+ 6601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:38.141399 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:38.141406 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:38.141446 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:38.145614 (+ 4168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:38.146363 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:38.146369 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:38.146372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:38.146516 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:38.147004 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:38.147009 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:38.147011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:38.147339 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:38.147482 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:38.149141 (+ 1659us) server_negotiation.cc:299] Negotiation successful
0620 23:11:38.149350 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":354,"threads_started":1}
I20260620 23:11:38.149921 18288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:38.120638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:44225 (local address 127.10.191.194:55421)
0620 23:11:38.124060 (+ 3422us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:38.124080 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:38.124208 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:38.125142 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:38.125158 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:38.129030 (+ 3872us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:38.129409 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:38.129424 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.131342 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.131350 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:38.135182 (+ 3832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:38.135194 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.136740 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.136751 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:38.137223 (+ 472us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:38.137851 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:38.137874 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:38.140595 (+ 2721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:38.144590 (+ 3995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:38.144600 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:38.144636 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:38.145268 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:38.145724 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:38.145729 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:38.145733 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:38.145982 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:38.146686 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:38.146694 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:38.147173 (+ 479us) client_negotiation.cc:769] Sending connection context
0620 23:11:38.148478 (+ 1305us) client_negotiation.cc:240] Negotiation successful
0620 23:11:38.148748 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3302,"mutex_wait_us":2083,"thread_start_us":3163,"threads_started":1}
I20260620 23:11:38.150224 18290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:38.124304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:42023 (local address 127.10.191.253:33657)
0620 23:11:38.132716 (+ 8412us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:38.132721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:38.132736 (+ 15us) server_negotiation.cc:407] Connection header received
0620 23:11:38.132782 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:38.132785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:38.132838 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:38.132967 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:38.134133 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.135209 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.137243 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.137929 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.138038 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:38.142409 (+ 4371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:38.142415 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:38.142419 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:38.142463 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:38.146199 (+ 3736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:38.147101 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:38.147107 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:38.147111 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:38.147273 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:38.147756 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:38.147762 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:38.147765 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:38.148123 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:38.148280 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:38.149445 (+ 1165us) server_negotiation.cc:299] Negotiation successful
0620 23:11:38.149708 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8306,"thread_start_us":8204,"threads_started":1}
I20260620 23:11:38.150241 18285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:38.119194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.10.191.194:45007)
0620 23:11:38.120069 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:38.120112 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:38.122852 (+ 2740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:38.125725 (+ 2873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:38.125732 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:38.127418 (+ 1686us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:38.128402 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:38.128434 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.131030 (+ 2596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.131038 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:38.134044 (+ 3006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:38.134059 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.134895 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.134903 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:38.135195 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:38.136004 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:38.136034 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:38.141189 (+ 5155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:38.145741 (+ 4552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:38.145744 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:38.145747 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:38.146235 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:38.146637 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:38.146643 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:38.146647 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:38.146846 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:38.147538 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:38.147542 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:38.147687 (+ 145us) client_negotiation.cc:769] Sending connection context
0620 23:11:38.147839 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 23:11:38.148403 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"mutex_wait_us":4,"thread_start_us":373,"threads_started":1}
I20260620 23:11:38.150610 18291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:38.126008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:33657 (local address 127.10.191.194:42023)
0620 23:11:38.126821 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:38.126844 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:38.126997 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:38.132983 (+ 5986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:38.132987 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:38.133573 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:38.133939 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:38.133954 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.135378 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.135383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:38.137072 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:38.137085 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.138321 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.138330 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:38.138486 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:38.139132 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:38.139154 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:38.142193 (+ 3039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:38.146362 (+ 4169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:38.146369 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:38.146375 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:38.146958 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:38.147410 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:38.147415 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:38.147420 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:38.147606 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:38.148359 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:38.148366 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:38.148502 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 23:11:38.148659 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 23:11:38.148946 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"mutex_wait_us":1,"thread_start_us":452,"threads_started":1}
I20260620 23:11:38.151680 18281 heartbeater.cc:347] Connected to a master server at 127.10.191.254:39679
I20260620 23:11:38.151804 18280 heartbeater.cc:347] Connected to a master server at 127.10.191.252:44225
I20260620 23:11:38.152033 18281 heartbeater.cc:464] Registering TS with master...
I20260620 23:11:38.152040 18280 heartbeater.cc:464] Registering TS with master...
I20260620 23:11:38.152814 18282 heartbeater.cc:347] Connected to a master server at 127.10.191.253:33657
I20260620 23:11:38.152936 18282 heartbeater.cc:464] Registering TS with master...
I20260620 23:11:38.153116 18280 heartbeater.cc:511] Master 127.10.191.252:44225 requested a full tablet report, sending...
I20260620 23:11:38.153393 18282 heartbeater.cc:511] Master 127.10.191.253:33657 requested a full tablet report, sending...
I20260620 23:11:38.154121 18281 heartbeater.cc:511] Master 127.10.191.254:39679 requested a full tablet report, sending...
I20260620 23:11:38.155072 17958 ts_manager.cc:194] Registered new tserver with Master: ab48ee8fa5634fc59ab80e8a1c3d5361 (127.10.191.194:35513)
I20260620 23:11:38.155135 17158 ts_manager.cc:194] Re-registered known tserver with Master: ab48ee8fa5634fc59ab80e8a1c3d5361 (127.10.191.194:35513)
I20260620 23:11:38.155274 17761 ts_manager.cc:194] Registered new tserver with Master: ab48ee8fa5634fc59ab80e8a1c3d5361 (127.10.191.194:35513)
I20260620 23:11:38.156277 17158 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.191.194@KRBTEST.COM'} at 127.10.191.194:45007
I20260620 23:11:38.157647 18287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:38.119749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50228 (local address 127.10.191.195:46823)
0620 23:11:38.134294 (+ 14545us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:38.134303 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:38.134330 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:11:38.134389 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:38.134397 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:38.134640 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:38.137035 (+ 2395us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:38.137309 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.139315 (+ 2006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.141117 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.141974 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.142778 (+ 804us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:38.148673 (+ 5895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:38.148687 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:38.148703 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:38.148740 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:38.153794 (+ 5054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:38.154512 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:38.154521 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:38.154524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:38.154699 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:38.156021 (+ 1322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:38.156026 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:38.156029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:38.156429 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:38.156745 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:38.157079 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 23:11:38.157267 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14375,"thread_start_us":14214,"threads_started":1}
I20260620 23:11:38.158777 17609 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:11:38.242168 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 17486
I20260620 23:11:38.258452 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.195:46823
--local_ip_for_outbound_sockets=127.10.191.195
--tserver_master_addrs=127.10.191.254:39679
--webserver_port=42537
--webserver_interface=127.10.191.195
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.195.keytab
--principal=oryx/127.10.191.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:33741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.191.254:39679,127.10.191.253:33657
--tserver_master_addrs=127.10.191.254:39679,127.10.191.253:33657,127.10.191.252:44225 with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:11:38.569720 18140 heartbeater.cc:503] Master 127.10.191.254:39679 was elected leader, sending a full tablet report...
W20260620 23:11:38.586618 18293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:38.587016 18293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:38.587199 18293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:38.594540 18293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:38.594683 18293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:38.594848 18293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.195
I20260620 23:11:38.604408 18293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:33741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.195.keytab
--principal=oryx/127.10.191.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.195:46823
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.191.195
--webserver_port=42537
--tserver_master_addrs=127.10.191.254:39679,127.10.191.253:33657,127.10.191.252:44225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.195
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:38.605772 18293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:38.607213 18293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:38.620616 18300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:38.620683 18302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:38.620743 18299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:38.621578 18293 server_base.cc:1034] running on GCE node
I20260620 23:11:38.622226 18293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:38.623196 18293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:38.624429 18293 hybrid_clock.cc:648] HybridClock initialized: now 1781997098624394 us; error 52 us; skew 500 ppm
I20260620 23:11:38.624749 18293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:11:38 dist-test-slave-jkbq krb5kdc[17113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997098, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.191.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:38.627740 18293 init.cc:377] Logged in from keytab as oryx/127.10.191.195@KRBTEST.COM (short username oryx)
I20260620 23:11:38.629086 18293 webserver.cc:460] Webserver started at http://127.10.191.195:42537/ using document root <none> and password file <none>
I20260620 23:11:38.629743 18293 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:38.629894 18293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:38.634986 18293 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:11:38.637519 18309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:38.638948 18293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 23:11:38.639081 18293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-2/wal
uuid: "25499f9be0e240fbbcf6d2864c0b51bf"
format_stamp: "Formatted at 2026-06-20 23:11:35 on dist-test-slave-jkbq"
server_key: "792a62c14c16cdf7af97cee7fa6a3fb0"
server_key_iv: "754acf90b461cc25f1060cf56ee1e121"
server_key_version: "encryptionkey@0"
I20260620 23:11:38.639983 18293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:38.659577 18293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:38.663868 18293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:38.664170 18293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:38.665086 18293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:11:38.666338 18293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:11:38.666420 18293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:38.666508 18293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:11:38.666561 18293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:38.699404 18293 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.195:46823
I20260620 23:11:38.699491 18421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.195:46823 every 8 connection(s)
I20260620 23:11:38.700819 18293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
I20260620 23:11:38.709964 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 18293
Jun 20 23:11:38 dist-test-slave-jkbq krb5kdc[17113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997098, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.191.195@KRBTEST.COM for oryx/127.10.191.253@KRBTEST.COM
Jun 20 23:11:38 dist-test-slave-jkbq krb5kdc[17113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997098, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.191.195@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM
Jun 20 23:11:38 dist-test-slave-jkbq krb5kdc[17113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997098, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.191.195@KRBTEST.COM for oryx/127.10.191.252@KRBTEST.COM
I20260620 23:11:38.730396 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:38.706193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:35667 (local address 127.10.191.252:44225)
0620 23:11:38.714192 (+ 7999us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:38.714198 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:38.714217 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:11:38.714274 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:38.714282 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:38.714370 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:38.715621 (+ 1251us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:38.715762 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.716852 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.718818 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.719507 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.720237 (+ 730us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:38.723530 (+ 3293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:38.723537 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:38.723543 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:38.723582 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:38.727240 (+ 3658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:38.727926 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:38.727931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:38.727934 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:38.728071 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:38.728563 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:38.728568 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:38.728571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:38.728905 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:38.729040 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:38.729827 (+ 787us) server_negotiation.cc:299] Negotiation successful
0620 23:11:38.730022 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7858,"thread_start_us":2292,"threads_started":1}
I20260620 23:11:38.731719 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:38.705966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:44225 (local address 127.10.191.195:35667)
0620 23:11:38.706562 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:38.706584 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:38.707981 (+ 1397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:38.714545 (+ 6564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:38.714555 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:38.715176 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:38.715476 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:38.715487 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.717018 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.717023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:38.718626 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:38.718640 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.719716 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.719722 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:38.719913 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:38.720465 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:38.720484 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:38.723353 (+ 2869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:38.727351 (+ 3998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:38.727359 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:38.727376 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:38.727798 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:38.728175 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:38.728180 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:38.728183 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:38.728456 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:38.729034 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:38.729042 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:38.729470 (+ 428us) client_negotiation.cc:769] Sending connection context
0620 23:11:38.729736 (+ 266us) client_negotiation.cc:240] Negotiation successful
0620 23:11:38.730230 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":294,"threads_started":1}
I20260620 23:11:38.733489 18422 heartbeater.cc:347] Connected to a master server at 127.10.191.252:44225
I20260620 23:11:38.733811 18422 heartbeater.cc:464] Registering TS with master...
I20260620 23:11:38.734568 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:38.707324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:33657 (local address 127.10.191.195:57683)
0620 23:11:38.707870 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:38.707887 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:38.708011 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:38.708903 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:38.708916 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:38.709857 (+ 941us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:38.710670 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:38.710687 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.714410 (+ 3723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.714418 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:38.716021 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:38.716033 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.717095 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.717100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:38.717288 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:38.719161 (+ 1873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:38.719183 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:38.723011 (+ 3828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:38.728188 (+ 5177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:38.728192 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:38.728195 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:38.729294 (+ 1099us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:38.731184 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:38.731189 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:38.731192 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:38.731351 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:38.733294 (+ 1943us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:38.733301 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:38.733448 (+ 147us) client_negotiation.cc:769] Sending connection context
0620 23:11:38.733661 (+ 213us) client_negotiation.cc:240] Negotiation successful
0620 23:11:38.734156 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":27,"thread_start_us":293,"threads_started":1}
I20260620 23:11:38.735005 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:38.706916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:57683 (local address 127.10.191.253:33657)
0620 23:11:38.707377 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:38.707384 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:38.707983 (+ 599us) server_negotiation.cc:407] Connection header received
0620 23:11:38.708278 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:38.708287 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:38.708395 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:38.708527 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:38.710840 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.711972 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.716156 (+ 4184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.716915 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.717053 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:38.723161 (+ 6108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:38.723167 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:38.723172 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:38.723216 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:38.727767 (+ 4551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:38.730809 (+ 3042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:38.730818 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:38.730824 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:38.731061 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:38.732051 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:38.732060 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:38.732066 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:38.733173 (+ 1107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:38.733357 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:38.733999 (+ 642us) server_negotiation.cc:299] Negotiation successful
0620 23:11:38.734298 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":236,"threads_started":1}
I20260620 23:11:38.734740 18422 heartbeater.cc:511] Master 127.10.191.252:44225 requested a full tablet report, sending...
I20260620 23:11:38.736994 18424 heartbeater.cc:347] Connected to a master server at 127.10.191.253:33657
I20260620 23:11:38.737154 18424 heartbeater.cc:464] Registering TS with master...
I20260620 23:11:38.737195 17958 ts_manager.cc:194] Registered new tserver with Master: 25499f9be0e240fbbcf6d2864c0b51bf (127.10.191.195:46823)
I20260620 23:11:38.737669 18424 heartbeater.cc:511] Master 127.10.191.253:33657 requested a full tablet report, sending...
I20260620 23:11:38.738785 17761 ts_manager.cc:194] Registered new tserver with Master: 25499f9be0e240fbbcf6d2864c0b51bf (127.10.191.195:46823)
I20260620 23:11:38.739032 18426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:38.703821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:51779 (local address 127.10.191.254:39679)
0620 23:11:38.704308 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:38.704314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:38.705616 (+ 1302us) server_negotiation.cc:407] Connection header received
0620 23:11:38.708424 (+ 2808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:38.708431 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:38.708528 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:38.708647 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:38.710794 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.711972 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.714420 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.715333 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.716287 (+ 954us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:38.722904 (+ 6617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:38.722915 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:38.722922 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:38.722986 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:38.733679 (+ 10693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:38.735486 (+ 1807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:38.735491 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:38.735495 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:38.735665 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:38.736079 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:38.736083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:38.736086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:38.737854 (+ 1768us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:38.738002 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:38.738421 (+ 419us) server_negotiation.cc:299] Negotiation successful
0620 23:11:38.738624 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":240,"threads_started":1}
I20260620 23:11:38.739053 18427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:38.704442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.10.191.195:51779)
0620 23:11:38.705464 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:38.705504 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:38.708313 (+ 2809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:38.708903 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:38.708916 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:38.709852 (+ 936us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:38.710617 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:38.710636 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.712160 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.712169 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:38.714228 (+ 2059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:38.714244 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.716378 (+ 2134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.716385 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:38.716663 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:38.717728 (+ 1065us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:38.717758 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:38.722649 (+ 4891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:38.733935 (+ 11286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:38.733939 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:38.733943 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:38.735234 (+ 1291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:38.735776 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:38.735783 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:38.735788 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:38.735938 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:38.737984 (+ 2046us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:38.737988 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:38.738160 (+ 172us) client_negotiation.cc:769] Sending connection context
0620 23:11:38.738298 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 23:11:38.738587 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"mutex_wait_us":37,"spinlock_wait_cycles":26496,"thread_start_us":630,"threads_started":1}
I20260620 23:11:38.740309 18423 heartbeater.cc:347] Connected to a master server at 127.10.191.254:39679
I20260620 23:11:38.740425 18423 heartbeater.cc:464] Registering TS with master...
I20260620 23:11:38.740778 18423 heartbeater.cc:511] Master 127.10.191.254:39679 requested a full tablet report, sending...
I20260620 23:11:38.741736 17158 ts_manager.cc:194] Re-registered known tserver with Master: 25499f9be0e240fbbcf6d2864c0b51bf (127.10.191.195:46823)
I20260620 23:11:38.742818 17158 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.191.195@KRBTEST.COM'} at 127.10.191.195:51779
I20260620 23:11:38.751531 18436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:38.714241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38240 (local address 127.10.191.254:39679)
0620 23:11:38.721722 (+ 7481us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:38.721728 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:38.722253 (+ 525us) server_negotiation.cc:407] Connection header received
0620 23:11:38.722560 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:38.722565 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:38.722657 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:38.722736 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:38.725836 (+ 3100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.727066 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.739444 (+ 12378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.740025 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.740134 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:38.745191 (+ 5057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:38.745200 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:38.745208 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:38.745246 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:38.748617 (+ 3371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:38.749464 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:38.749469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:38.749473 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:38.749626 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:38.750065 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:38.750070 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:38.750073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:38.750314 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:38.750538 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:38.750942 (+ 404us) server_negotiation.cc:299] Negotiation successful
0620 23:11:38.751164 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7365,"thread_start_us":7267,"threads_started":1}
I20260620 23:11:38.753428 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:38.722570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42458 (local address 127.10.191.253:33657)
0620 23:11:38.735415 (+ 12845us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:38.735422 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:38.736248 (+ 826us) server_negotiation.cc:407] Connection header received
0620 23:11:38.736296 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:38.736300 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:38.736375 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:38.736918 (+ 543us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:38.740347 (+ 3429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.741393 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.743343 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.743960 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.744054 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:38.747283 (+ 3229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:38.747288 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:38.747291 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:38.747320 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:38.750680 (+ 3360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:38.751494 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:38.751499 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:38.751502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:38.751641 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:38.752114 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:38.752119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:38.752122 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:38.752411 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:38.752663 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:38.753015 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 23:11:38.753196 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12685,"thread_start_us":12875,"threads_started":1}
I20260620 23:11:38.756995 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:38.736430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38364 (local address 127.10.191.252:44225)
0620 23:11:38.737668 (+ 1238us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:38.737673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:38.740073 (+ 2400us) server_negotiation.cc:407] Connection header received
0620 23:11:38.740330 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:38.740336 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:38.740405 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:38.740502 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:38.742714 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.743862 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.746940 (+ 3078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.747605 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.747745 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:38.751088 (+ 3343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:38.751095 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:38.751100 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:38.751137 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:38.754448 (+ 3311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:38.755197 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:38.755205 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:38.755208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:38.755343 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:38.755766 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:38.755770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:38.755772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:38.756025 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:38.756218 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:38.756568 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 23:11:38.756732 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1131}
I20260620 23:11:38.756139 17158 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:11:38.764034 17158 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:11:38.807420 18452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:38.782398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.195:46823 (local address 127.0.0.1:50230)
0620 23:11:38.783031 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:38.783052 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:38.789417 (+ 6365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:38.800114 (+ 10697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:38.800119 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:38.800145 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:11:38.800429 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:38.800438 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.802503 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.802509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:38.804826 (+ 2317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:38.804840 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.806326 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.806335 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:38.806555 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:38.806593 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 23:11:38.806694 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 23:11:38.806951 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":77,"thread_start_us":330,"threads_started":1}
I20260620 23:11:38.808409 18453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:38.782777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50230 (local address 127.10.191.195:46823)
0620 23:11:38.799308 (+ 16531us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:38.799320 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:38.799366 (+ 46us) server_negotiation.cc:407] Connection header received
0620 23:11:38.799500 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:38.799512 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:38.799845 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:38.800107 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:11:38.800695 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.802368 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.805155 (+ 2787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.806090 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.806219 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:38.806344 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:38.807925 (+ 1581us) server_negotiation.cc:299] Negotiation successful
0620 23:11:38.808095 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16306,"thread_start_us":16122,"threads_started":1}
I20260620 23:11:38.811638 18455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:38.783299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:40211 (local address 127.0.0.1:53390)
0620 23:11:38.801062 (+ 17763us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:38.801090 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:38.801206 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:38.802267 (+ 1061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:38.802275 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:38.802311 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:11:38.802987 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:38.803003 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.804856 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.804862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:38.808735 (+ 3873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:38.808749 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.810717 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.810721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:38.810950 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:38.810983 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 23:11:38.811059 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 23:11:38.811241 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17634,"mutex_wait_us":52,"thread_start_us":23127,"threads_started":1}
I20260620 23:11:38.812345 18457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:38.783696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53390 (local address 127.10.191.193:40211)
0620 23:11:38.801668 (+ 17972us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:38.801677 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:38.801703 (+ 26us) server_negotiation.cc:407] Connection header received
0620 23:11:38.801788 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:38.801796 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:38.802022 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:38.802247 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:11:38.803202 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.804690 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.808892 (+ 4202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.810255 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.810419 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:38.810601 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:38.811875 (+ 1274us) server_negotiation.cc:299] Negotiation successful
0620 23:11:38.812038 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17811,"thread_start_us":17698,"threads_started":1}
I20260620 23:11:38.812379 18357 tablet_service.cc:1467] Processing CreateTablet for tablet 0ef2b01b6ee2460eacba7cb74752353f (DEFAULT_TABLE table=test-table [id=adedcd5ffe0d47c89c1904c06a64ee20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:11:38.813844 18357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ef2b01b6ee2460eacba7cb74752353f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:11:38.815780 18456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:38.783591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56502 (local address 127.10.191.194:35513)
0620 23:11:38.796025 (+ 12434us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:38.796038 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:38.807017 (+ 10979us) server_negotiation.cc:407] Connection header received
0620 23:11:38.807136 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:38.807146 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:38.807368 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:38.807692 (+ 324us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:11:38.809149 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.810934 (+ 1785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.813087 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.814764 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.814973 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:38.815135 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:38.815356 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 23:11:38.815513 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12202,"thread_start_us":12433,"threads_started":1}
I20260620 23:11:38.815852 18454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:38.783299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.194:35513 (local address 127.0.0.1:56502)
0620 23:11:38.806721 (+ 23422us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:38.806739 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:38.806857 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:38.808632 (+ 1775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:38.808637 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:38.808668 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:11:38.808987 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:38.808995 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.811069 (+ 2074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.811073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:38.812910 (+ 1837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:38.812921 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.814990 (+ 2069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.814996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:38.815162 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:38.815205 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 23:11:38.815304 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 23:11:38.815417 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23295,"mutex_wait_us":157,"thread_start_us":17344,"threads_started":1}
I20260620 23:11:38.817103 18074 tablet_service.cc:1467] Processing CreateTablet for tablet 0ef2b01b6ee2460eacba7cb74752353f (DEFAULT_TABLE table=test-table [id=adedcd5ffe0d47c89c1904c06a64ee20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:11:38.818195 18074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ef2b01b6ee2460eacba7cb74752353f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:11:38.818517 18215 tablet_service.cc:1467] Processing CreateTablet for tablet 0ef2b01b6ee2460eacba7cb74752353f (DEFAULT_TABLE table=test-table [id=adedcd5ffe0d47c89c1904c06a64ee20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:11:38.819676 18215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ef2b01b6ee2460eacba7cb74752353f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:11:38.837255 18459 tablet_bootstrap.cc:492] T 0ef2b01b6ee2460eacba7cb74752353f P ab48ee8fa5634fc59ab80e8a1c3d5361: Bootstrap starting.
I20260620 23:11:38.841576 18459 tablet_bootstrap.cc:654] T 0ef2b01b6ee2460eacba7cb74752353f P ab48ee8fa5634fc59ab80e8a1c3d5361: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:38.842675 18460 tablet_bootstrap.cc:492] T 0ef2b01b6ee2460eacba7cb74752353f P 25499f9be0e240fbbcf6d2864c0b51bf: Bootstrap starting.
I20260620 23:11:38.844874 18458 tablet_bootstrap.cc:492] T 0ef2b01b6ee2460eacba7cb74752353f P f3868efeb1fa4926a2fa269532dddbab: Bootstrap starting.
I20260620 23:11:38.847136 18459 log.cc:824] T 0ef2b01b6ee2460eacba7cb74752353f P ab48ee8fa5634fc59ab80e8a1c3d5361: Log is configured to *not* fsync() on all Append() calls
I20260620 23:11:38.848287 18458 tablet_bootstrap.cc:654] T 0ef2b01b6ee2460eacba7cb74752353f P f3868efeb1fa4926a2fa269532dddbab: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:38.848489 18460 tablet_bootstrap.cc:654] T 0ef2b01b6ee2460eacba7cb74752353f P 25499f9be0e240fbbcf6d2864c0b51bf: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:38.849642 18460 log.cc:824] T 0ef2b01b6ee2460eacba7cb74752353f P 25499f9be0e240fbbcf6d2864c0b51bf: Log is configured to *not* fsync() on all Append() calls
I20260620 23:11:38.849630 18458 log.cc:824] T 0ef2b01b6ee2460eacba7cb74752353f P f3868efeb1fa4926a2fa269532dddbab: Log is configured to *not* fsync() on all Append() calls
I20260620 23:11:38.856109 18459 tablet_bootstrap.cc:492] T 0ef2b01b6ee2460eacba7cb74752353f P ab48ee8fa5634fc59ab80e8a1c3d5361: No bootstrap required, opened a new log
I20260620 23:11:38.856387 18459 ts_tablet_manager.cc:1397] T 0ef2b01b6ee2460eacba7cb74752353f P ab48ee8fa5634fc59ab80e8a1c3d5361: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.009s
I20260620 23:11:38.857311 18460 tablet_bootstrap.cc:492] T 0ef2b01b6ee2460eacba7cb74752353f P 25499f9be0e240fbbcf6d2864c0b51bf: No bootstrap required, opened a new log
I20260620 23:11:38.857710 18460 ts_tablet_manager.cc:1397] T 0ef2b01b6ee2460eacba7cb74752353f P 25499f9be0e240fbbcf6d2864c0b51bf: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.006s
I20260620 23:11:38.865226 18459 raft_consensus.cc:348] T 0ef2b01b6ee2460eacba7cb74752353f P ab48ee8fa5634fc59ab80e8a1c3d5361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25499f9be0e240fbbcf6d2864c0b51bf" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46823 } } peers { permanent_uuid: "f3868efeb1fa4926a2fa269532dddbab" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40211 } } peers { permanent_uuid: "ab48ee8fa5634fc59ab80e8a1c3d5361" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 35513 } }
I20260620 23:11:38.865592 18459 raft_consensus.cc:374] T 0ef2b01b6ee2460eacba7cb74752353f P ab48ee8fa5634fc59ab80e8a1c3d5361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:38.865700 18459 raft_consensus.cc:729] T 0ef2b01b6ee2460eacba7cb74752353f P ab48ee8fa5634fc59ab80e8a1c3d5361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab48ee8fa5634fc59ab80e8a1c3d5361, State: Initialized, Role: FOLLOWER
I20260620 23:11:38.866320 18459 consensus_queue.cc:260] T 0ef2b01b6ee2460eacba7cb74752353f P ab48ee8fa5634fc59ab80e8a1c3d5361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25499f9be0e240fbbcf6d2864c0b51bf" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46823 } } peers { permanent_uuid: "f3868efeb1fa4926a2fa269532dddbab" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40211 } } peers { permanent_uuid: "ab48ee8fa5634fc59ab80e8a1c3d5361" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 35513 } }
I20260620 23:11:38.870038 18459 ts_tablet_manager.cc:1428] T 0ef2b01b6ee2460eacba7cb74752353f P ab48ee8fa5634fc59ab80e8a1c3d5361: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260620 23:11:38.870306 18281 heartbeater.cc:503] Master 127.10.191.254:39679 was elected leader, sending a full tablet report...
I20260620 23:11:38.871064 18458 tablet_bootstrap.cc:492] T 0ef2b01b6ee2460eacba7cb74752353f P f3868efeb1fa4926a2fa269532dddbab: No bootstrap required, opened a new log
I20260620 23:11:38.871414 18458 ts_tablet_manager.cc:1397] T 0ef2b01b6ee2460eacba7cb74752353f P f3868efeb1fa4926a2fa269532dddbab: Time spent bootstrapping tablet: real 0.027s user 0.012s sys 0.013s
I20260620 23:11:38.871924 18460 raft_consensus.cc:348] T 0ef2b01b6ee2460eacba7cb74752353f P 25499f9be0e240fbbcf6d2864c0b51bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25499f9be0e240fbbcf6d2864c0b51bf" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46823 } } peers { permanent_uuid: "f3868efeb1fa4926a2fa269532dddbab" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40211 } } peers { permanent_uuid: "ab48ee8fa5634fc59ab80e8a1c3d5361" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 35513 } }
I20260620 23:11:38.872386 18460 raft_consensus.cc:374] T 0ef2b01b6ee2460eacba7cb74752353f P 25499f9be0e240fbbcf6d2864c0b51bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:38.872496 18460 raft_consensus.cc:729] T 0ef2b01b6ee2460eacba7cb74752353f P 25499f9be0e240fbbcf6d2864c0b51bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25499f9be0e240fbbcf6d2864c0b51bf, State: Initialized, Role: FOLLOWER
I20260620 23:11:38.873121 18460 consensus_queue.cc:260] T 0ef2b01b6ee2460eacba7cb74752353f P 25499f9be0e240fbbcf6d2864c0b51bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25499f9be0e240fbbcf6d2864c0b51bf" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46823 } } peers { permanent_uuid: "f3868efeb1fa4926a2fa269532dddbab" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40211 } } peers { permanent_uuid: "ab48ee8fa5634fc59ab80e8a1c3d5361" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 35513 } }
I20260620 23:11:38.876282 18460 ts_tablet_manager.cc:1428] T 0ef2b01b6ee2460eacba7cb74752353f P 25499f9be0e240fbbcf6d2864c0b51bf: Time spent starting tablet: real 0.018s user 0.012s sys 0.005s
I20260620 23:11:38.876528 18423 heartbeater.cc:503] Master 127.10.191.254:39679 was elected leader, sending a full tablet report...
I20260620 23:11:38.881590 18458 raft_consensus.cc:348] T 0ef2b01b6ee2460eacba7cb74752353f P f3868efeb1fa4926a2fa269532dddbab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25499f9be0e240fbbcf6d2864c0b51bf" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46823 } } peers { permanent_uuid: "f3868efeb1fa4926a2fa269532dddbab" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40211 } } peers { permanent_uuid: "ab48ee8fa5634fc59ab80e8a1c3d5361" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 35513 } }
I20260620 23:11:38.881964 18458 raft_consensus.cc:374] T 0ef2b01b6ee2460eacba7cb74752353f P f3868efeb1fa4926a2fa269532dddbab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:38.882088 18458 raft_consensus.cc:729] T 0ef2b01b6ee2460eacba7cb74752353f P f3868efeb1fa4926a2fa269532dddbab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3868efeb1fa4926a2fa269532dddbab, State: Initialized, Role: FOLLOWER
I20260620 23:11:38.882709 18458 consensus_queue.cc:260] T 0ef2b01b6ee2460eacba7cb74752353f P f3868efeb1fa4926a2fa269532dddbab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25499f9be0e240fbbcf6d2864c0b51bf" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46823 } } peers { permanent_uuid: "f3868efeb1fa4926a2fa269532dddbab" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40211 } } peers { permanent_uuid: "ab48ee8fa5634fc59ab80e8a1c3d5361" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 35513 } }
I20260620 23:11:38.884333 18458 ts_tablet_manager.cc:1428] T 0ef2b01b6ee2460eacba7cb74752353f P f3868efeb1fa4926a2fa269532dddbab: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260620 23:11:38.918503 18465 raft_consensus.cc:482] T 0ef2b01b6ee2460eacba7cb74752353f P 25499f9be0e240fbbcf6d2864c0b51bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:11:38.918823 18465 raft_consensus.cc:504] T 0ef2b01b6ee2460eacba7cb74752353f P 25499f9be0e240fbbcf6d2864c0b51bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25499f9be0e240fbbcf6d2864c0b51bf" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46823 } } peers { permanent_uuid: "f3868efeb1fa4926a2fa269532dddbab" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40211 } } peers { permanent_uuid: "ab48ee8fa5634fc59ab80e8a1c3d5361" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 35513 } }
I20260620 23:11:38.920509 18465 leader_election.cc:288] T 0ef2b01b6ee2460eacba7cb74752353f P 25499f9be0e240fbbcf6d2864c0b51bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3868efeb1fa4926a2fa269532dddbab (127.10.191.193:40211), ab48ee8fa5634fc59ab80e8a1c3d5361 (127.10.191.194:35513)
I20260620 23:11:38.928459 18456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:38.920910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:53759 (local address 127.10.191.194:35513)
0620 23:11:38.921088 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:38.921094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:38.921171 (+ 77us) server_negotiation.cc:407] Connection header received
0620 23:11:38.921408 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:38.921414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:38.921491 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:38.921628 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:11:38.922175 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.923339 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.926041 (+ 2702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.927313 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.927476 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:38.927536 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:38.928084 (+ 548us) server_negotiation.cc:299] Negotiation successful
0620 23:11:38.928218 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 23:11:38.928459 18457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:38.920896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:37485 (local address 127.10.191.193:40211)
0620 23:11:38.921103 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:38.921111 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:38.921330 (+ 219us) server_negotiation.cc:407] Connection header received
0620 23:11:38.921457 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:38.921462 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:38.921534 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:38.921623 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:11:38.922175 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.923338 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.926040 (+ 2702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.927313 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.927464 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:38.927536 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:38.928084 (+ 548us) server_negotiation.cc:299] Negotiation successful
0620 23:11:38.928212 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:11:38.928472 18427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:38.920776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.194:35513 (local address 127.10.191.195:53759)
0620 23:11:38.921068 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:38.921088 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:38.921233 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:38.921649 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:38.921654 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:38.921692 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:11:38.922025 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:38.922036 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.923473 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.923477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:38.925862 (+ 2385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:38.925876 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.927465 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.927472 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:38.927740 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:38.927824 (+ 84us) client_negotiation.cc:769] Sending connection context
0620 23:11:38.927925 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 23:11:38.928060 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":298,"spinlock_wait_cycles":9600}
I20260620 23:11:38.928524 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:38.920954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:40211 (local address 127.10.191.195:37485)
0620 23:11:38.921220 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:38.921235 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:38.921338 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:38.921649 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:38.921654 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:38.921694 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:11:38.922027 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:38.922036 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.923471 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.923476 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:38.925862 (+ 2386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:38.925876 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.927465 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.927472 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:38.927742 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:38.927820 (+ 78us) client_negotiation.cc:769] Sending connection context
0620 23:11:38.927958 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 23:11:38.928064 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":312,"spinlock_wait_cycles":15104}
I20260620 23:11:38.929533 18094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ef2b01b6ee2460eacba7cb74752353f" candidate_uuid: "25499f9be0e240fbbcf6d2864c0b51bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3868efeb1fa4926a2fa269532dddbab" is_pre_election: true
I20260620 23:11:38.929541 18235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ef2b01b6ee2460eacba7cb74752353f" candidate_uuid: "25499f9be0e240fbbcf6d2864c0b51bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab48ee8fa5634fc59ab80e8a1c3d5361" is_pre_election: true
I20260620 23:11:38.930068 18235 raft_consensus.cc:2454] T 0ef2b01b6ee2460eacba7cb74752353f P ab48ee8fa5634fc59ab80e8a1c3d5361 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25499f9be0e240fbbcf6d2864c0b51bf in term 0.
I20260620 23:11:38.930068 18094 raft_consensus.cc:2454] T 0ef2b01b6ee2460eacba7cb74752353f P f3868efeb1fa4926a2fa269532dddbab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25499f9be0e240fbbcf6d2864c0b51bf in term 0.
I20260620 23:11:38.930783 18313 leader_election.cc:302] T 0ef2b01b6ee2460eacba7cb74752353f P 25499f9be0e240fbbcf6d2864c0b51bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 25499f9be0e240fbbcf6d2864c0b51bf, ab48ee8fa5634fc59ab80e8a1c3d5361, f3868efeb1fa4926a2fa269532dddbab; no voters:
I20260620 23:11:38.931270 18465 raft_consensus.cc:2781] T 0ef2b01b6ee2460eacba7cb74752353f P 25499f9be0e240fbbcf6d2864c0b51bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:11:38.931419 18465 raft_consensus.cc:482] T 0ef2b01b6ee2460eacba7cb74752353f P 25499f9be0e240fbbcf6d2864c0b51bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:11:38.931525 18465 raft_consensus.cc:3037] T 0ef2b01b6ee2460eacba7cb74752353f P 25499f9be0e240fbbcf6d2864c0b51bf [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:38.933625 18465 raft_consensus.cc:504] T 0ef2b01b6ee2460eacba7cb74752353f P 25499f9be0e240fbbcf6d2864c0b51bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25499f9be0e240fbbcf6d2864c0b51bf" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46823 } } peers { permanent_uuid: "f3868efeb1fa4926a2fa269532dddbab" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40211 } } peers { permanent_uuid: "ab48ee8fa5634fc59ab80e8a1c3d5361" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 35513 } }
I20260620 23:11:38.934154 18465 leader_election.cc:288] T 0ef2b01b6ee2460eacba7cb74752353f P 25499f9be0e240fbbcf6d2864c0b51bf [CANDIDATE]: Term 1 election: Requested vote from peers f3868efeb1fa4926a2fa269532dddbab (127.10.191.193:40211), ab48ee8fa5634fc59ab80e8a1c3d5361 (127.10.191.194:35513)
I20260620 23:11:38.934607 18235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ef2b01b6ee2460eacba7cb74752353f" candidate_uuid: "25499f9be0e240fbbcf6d2864c0b51bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab48ee8fa5634fc59ab80e8a1c3d5361"
I20260620 23:11:38.934608 18094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ef2b01b6ee2460eacba7cb74752353f" candidate_uuid: "25499f9be0e240fbbcf6d2864c0b51bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3868efeb1fa4926a2fa269532dddbab"
I20260620 23:11:38.934808 18235 raft_consensus.cc:3037] T 0ef2b01b6ee2460eacba7cb74752353f P ab48ee8fa5634fc59ab80e8a1c3d5361 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:38.934808 18094 raft_consensus.cc:3037] T 0ef2b01b6ee2460eacba7cb74752353f P f3868efeb1fa4926a2fa269532dddbab [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:38.937991 18094 raft_consensus.cc:2454] T 0ef2b01b6ee2460eacba7cb74752353f P f3868efeb1fa4926a2fa269532dddbab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25499f9be0e240fbbcf6d2864c0b51bf in term 1.
I20260620 23:11:38.937991 18235 raft_consensus.cc:2454] T 0ef2b01b6ee2460eacba7cb74752353f P ab48ee8fa5634fc59ab80e8a1c3d5361 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25499f9be0e240fbbcf6d2864c0b51bf in term 1.
I20260620 23:11:38.938453 18311 leader_election.cc:302] T 0ef2b01b6ee2460eacba7cb74752353f P 25499f9be0e240fbbcf6d2864c0b51bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 25499f9be0e240fbbcf6d2864c0b51bf, ab48ee8fa5634fc59ab80e8a1c3d5361, f3868efeb1fa4926a2fa269532dddbab; no voters:
I20260620 23:11:38.938820 18465 raft_consensus.cc:2781] T 0ef2b01b6ee2460eacba7cb74752353f P 25499f9be0e240fbbcf6d2864c0b51bf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:11:38.939287 18465 raft_consensus.cc:686] T 0ef2b01b6ee2460eacba7cb74752353f P 25499f9be0e240fbbcf6d2864c0b51bf [term 1 LEADER]: Becoming Leader. State: Replica: 25499f9be0e240fbbcf6d2864c0b51bf, State: Running, Role: LEADER
I20260620 23:11:38.939833 18465 consensus_queue.cc:237] T 0ef2b01b6ee2460eacba7cb74752353f P 25499f9be0e240fbbcf6d2864c0b51bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25499f9be0e240fbbcf6d2864c0b51bf" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46823 } } peers { permanent_uuid: "f3868efeb1fa4926a2fa269532dddbab" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40211 } } peers { permanent_uuid: "ab48ee8fa5634fc59ab80e8a1c3d5361" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 35513 } }
I20260620 23:11:38.945150 17158 catalog_manager.cc:5495] T 0ef2b01b6ee2460eacba7cb74752353f P 25499f9be0e240fbbcf6d2864c0b51bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 25499f9be0e240fbbcf6d2864c0b51bf (127.10.191.195). New cstate: current_term: 1 leader_uuid: "25499f9be0e240fbbcf6d2864c0b51bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25499f9be0e240fbbcf6d2864c0b51bf" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3868efeb1fa4926a2fa269532dddbab" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab48ee8fa5634fc59ab80e8a1c3d5361" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 35513 } health_report { overall_health: UNKNOWN } } }
W20260620 23:11:38.955364 18425 tablet.cc:2369] T 0ef2b01b6ee2460eacba7cb74752353f P 25499f9be0e240fbbcf6d2864c0b51bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:11:38.981940 18453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:38.976930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50234 (local address 127.10.191.195:46823)
0620 23:11:38.977175 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:38.977188 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:38.977205 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:11:38.977387 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:38.977391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:38.977472 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:38.977590 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:11:38.977912 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.978589 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.979924 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:38.980316 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:38.980404 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:38.980898 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:11:38.981389 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:11:38.981489 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:38.981594 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 23:11:38.981652 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 23:11:38.990185 18094 raft_consensus.cc:1261] T 0ef2b01b6ee2460eacba7cb74752353f P f3868efeb1fa4926a2fa269532dddbab [term 1 FOLLOWER]: Refusing update from remote peer 25499f9be0e240fbbcf6d2864c0b51bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:11:38.990219 18235 raft_consensus.cc:1261] T 0ef2b01b6ee2460eacba7cb74752353f P ab48ee8fa5634fc59ab80e8a1c3d5361 [term 1 FOLLOWER]: Refusing update from remote peer 25499f9be0e240fbbcf6d2864c0b51bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:11:38.991251 18467 consensus_queue.cc:1035] T 0ef2b01b6ee2460eacba7cb74752353f P 25499f9be0e240fbbcf6d2864c0b51bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3868efeb1fa4926a2fa269532dddbab" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:11:38.991629 18465 consensus_queue.cc:1035] T 0ef2b01b6ee2460eacba7cb74752353f P 25499f9be0e240fbbcf6d2864c0b51bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab48ee8fa5634fc59ab80e8a1c3d5361" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 35513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:11:39.003023 18470 mvcc.cc:205] Tried to move back new op lower bound from 7299060117452607488 to 7299060117262077952. Current Snapshot: MvccSnapshot[applied={T|T < 7299060117452607488}]
I20260620 23:11:39.003660 18472 mvcc.cc:205] Tried to move back new op lower bound from 7299060117452607488 to 7299060117262077952. Current Snapshot: MvccSnapshot[applied={T|T < 7299060117452607488}]
I20260620 23:11:39.010758 18471 mvcc.cc:205] Tried to move back new op lower bound from 7299060117452607488 to 7299060117262077952. Current Snapshot: MvccSnapshot[applied={T|T < 7299060117452607488}]
I20260620 23:11:39.027045 17158 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:11:39.027288 17158 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:11:39.034747 17158 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c8e2e42e0532441a97821bf64376c25d: Sending DeleteTablet for 3 replicas of tablet 0ef2b01b6ee2460eacba7cb74752353f
I20260620 23:11:39.035825 18357 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ef2b01b6ee2460eacba7cb74752353f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:11:39 UTC) from {username='oryx', principal='oryx/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:50230
I20260620 23:11:39.036062 18215 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ef2b01b6ee2460eacba7cb74752353f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:11:39 UTC) from {username='oryx', principal='oryx/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:56502
I20260620 23:11:39.037403 18074 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ef2b01b6ee2460eacba7cb74752353f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:11:39 UTC) from {username='oryx', principal='oryx/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:53390
I20260620 23:11:39.047153 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 17126
I20260620 23:11:39.048338 17188 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:11:39.053717 18479 tablet_replica.cc:330] T 0ef2b01b6ee2460eacba7cb74752353f P 25499f9be0e240fbbcf6d2864c0b51bf: stopping tablet replica
I20260620 23:11:39.054315 18479 raft_consensus.cc:2229] T 0ef2b01b6ee2460eacba7cb74752353f P 25499f9be0e240fbbcf6d2864c0b51bf [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:11:39.055092 18479 raft_consensus.cc:2258] T 0ef2b01b6ee2460eacba7cb74752353f P 25499f9be0e240fbbcf6d2864c0b51bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:11:39.055382 18481 tablet_replica.cc:330] T 0ef2b01b6ee2460eacba7cb74752353f P f3868efeb1fa4926a2fa269532dddbab: stopping tablet replica
I20260620 23:11:39.055848 18481 raft_consensus.cc:2229] T 0ef2b01b6ee2460eacba7cb74752353f P f3868efeb1fa4926a2fa269532dddbab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:11:39.056401 18481 raft_consensus.cc:2258] T 0ef2b01b6ee2460eacba7cb74752353f P f3868efeb1fa4926a2fa269532dddbab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:11:39.058813 18481 ts_tablet_manager.cc:1905] T 0ef2b01b6ee2460eacba7cb74752353f P f3868efeb1fa4926a2fa269532dddbab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:11:39.062459 18480 tablet_replica.cc:330] T 0ef2b01b6ee2460eacba7cb74752353f P ab48ee8fa5634fc59ab80e8a1c3d5361: stopping tablet replica
I20260620 23:11:39.064137 18480 raft_consensus.cc:2229] T 0ef2b01b6ee2460eacba7cb74752353f P ab48ee8fa5634fc59ab80e8a1c3d5361 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:11:39.064572 18479 ts_tablet_manager.cc:1905] T 0ef2b01b6ee2460eacba7cb74752353f P 25499f9be0e240fbbcf6d2864c0b51bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:11:39.064754 18480 raft_consensus.cc:2258] T 0ef2b01b6ee2460eacba7cb74752353f P ab48ee8fa5634fc59ab80e8a1c3d5361 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:11:39.069773 18480 ts_tablet_manager.cc:1905] T 0ef2b01b6ee2460eacba7cb74752353f P ab48ee8fa5634fc59ab80e8a1c3d5361: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:11:39.075706 18481 ts_tablet_manager.cc:1918] T 0ef2b01b6ee2460eacba7cb74752353f P f3868efeb1fa4926a2fa269532dddbab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:11:39.075963 18481 log.cc:1196] T 0ef2b01b6ee2460eacba7cb74752353f P f3868efeb1fa4926a2fa269532dddbab: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-0/wal/wals/0ef2b01b6ee2460eacba7cb74752353f
I20260620 23:11:39.076611 18481 ts_tablet_manager.cc:1939] T 0ef2b01b6ee2460eacba7cb74752353f P f3868efeb1fa4926a2fa269532dddbab: Deleting consensus metadata
I20260620 23:11:39.080803 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:39.040890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42462 (local address 127.10.191.253:33657)
0620 23:11:39.041043 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:39.041048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:39.045723 (+ 4675us) server_negotiation.cc:407] Connection header received
0620 23:11:39.046703 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:39.046710 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:39.046790 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:39.046885 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:39.055580 (+ 8695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:39.056701 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:39.059354 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:39.059986 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:39.060093 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:39.072719 (+ 12626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:39.072727 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:39.072732 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:39.072779 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:39.076550 (+ 3771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:39.077488 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:39.077493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:39.077498 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:39.077647 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:39.078099 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:39.078104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:39.078106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:39.078487 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:39.078715 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:39.080163 (+ 1448us) server_negotiation.cc:299] Negotiation successful
0620 23:11:39.080387 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 23:11:39.080854 18480 ts_tablet_manager.cc:1918] T 0ef2b01b6ee2460eacba7cb74752353f P ab48ee8fa5634fc59ab80e8a1c3d5361: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:11:39.081434 18480 log.cc:1196] T 0ef2b01b6ee2460eacba7cb74752353f P ab48ee8fa5634fc59ab80e8a1c3d5361: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-1/wal/wals/0ef2b01b6ee2460eacba7cb74752353f
I20260620 23:11:39.082041 18480 ts_tablet_manager.cc:1939] T 0ef2b01b6ee2460eacba7cb74752353f P ab48ee8fa5634fc59ab80e8a1c3d5361: Deleting consensus metadata
I20260620 23:11:39.086990 18479 ts_tablet_manager.cc:1918] T 0ef2b01b6ee2460eacba7cb74752353f P 25499f9be0e240fbbcf6d2864c0b51bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:11:39.087168 18479 log.cc:1196] T 0ef2b01b6ee2460eacba7cb74752353f P 25499f9be0e240fbbcf6d2864c0b51bf: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-2/wal/wals/0ef2b01b6ee2460eacba7cb74752353f
I20260620 23:11:39.087786 18479 ts_tablet_manager.cc:1939] T 0ef2b01b6ee2460eacba7cb74752353f P 25499f9be0e240fbbcf6d2864c0b51bf: Deleting consensus metadata
I20260620 23:11:39.097457 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:39.045924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38368 (local address 127.10.191.252:44225)
0620 23:11:39.046136 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:39.046144 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:39.082554 (+ 36410us) server_negotiation.cc:407] Connection header received
0620 23:11:39.082847 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:39.082852 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:39.082959 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:39.083030 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:39.084698 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:39.085790 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:39.087765 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:39.088421 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:39.088521 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:39.091768 (+ 3247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:39.091775 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:39.091779 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:39.091830 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:39.094771 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:39.095415 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:39.095420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:39.095424 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:39.095561 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:39.095971 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:39.095976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:39.095980 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:39.096250 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:39.096503 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:39.096813 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 23:11:39.097016 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 23:11:39.229775 17142 catalog_manager.cc:4841] TS 25499f9be0e240fbbcf6d2864c0b51bf (127.10.191.195:46823): tablet 0ef2b01b6ee2460eacba7cb74752353f (table test-table [id=adedcd5ffe0d47c89c1904c06a64ee20]) successfully deleted
I20260620 23:11:39.230298 17143 catalog_manager.cc:4841] TS f3868efeb1fa4926a2fa269532dddbab (127.10.191.193:40211): tablet 0ef2b01b6ee2460eacba7cb74752353f (table test-table [id=adedcd5ffe0d47c89c1904c06a64ee20]) successfully deleted
I20260620 23:11:39.230499 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 17126
I20260620 23:11:39.230526 17145 catalog_manager.cc:4841] TS ab48ee8fa5634fc59ab80e8a1c3d5361 (127.10.191.194:35513): tablet 0ef2b01b6ee2460eacba7cb74752353f (table test-table [id=adedcd5ffe0d47c89c1904c06a64ee20]) successfully deleted
I20260620 23:11:39.277420 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:39.261496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42476 (local address 127.10.191.253:33657)
0620 23:11:39.261698 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:39.261704 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:39.261725 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:11:39.261914 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:39.261920 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:39.262000 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:39.262108 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:39.264262 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:39.265389 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:39.267196 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:39.267616 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:39.267756 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:39.271359 (+ 3603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:39.271365 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:39.271370 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:39.271411 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:39.274677 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:39.275468 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:39.275476 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:39.275481 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:39.275634 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:39.276013 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:39.276016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:39.276018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:39.276299 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:39.276551 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:39.276866 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 23:11:39.277078 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 23:11:39.279134 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:39.262671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38378 (local address 127.10.191.252:44225)
0620 23:11:39.262886 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:39.262892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:39.263218 (+ 326us) server_negotiation.cc:407] Connection header received
0620 23:11:39.263496 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:39.263503 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:39.263593 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:39.263707 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:39.265367 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:39.266517 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:39.268505 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:39.268961 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:39.269064 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:39.272485 (+ 3421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:39.272491 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:39.272496 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:39.272536 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:39.275618 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:39.276653 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:39.276658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:39.276661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:39.276807 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:39.277229 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:39.277234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:39.277236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:39.277608 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:39.277819 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:39.278388 (+ 569us) server_negotiation.cc:299] Negotiation successful
0620 23:11:39.278788 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:11:40.019773 18490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:40.019053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.10.191.194:56575)
0620 23:11:40.019475 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:40.019612 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":208,"threads_started":1}
W20260620 23:11:40.020244 18281 heartbeater.cc:644] Failed to heartbeat to 127.10.191.254:39679 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
I20260620 23:11:40.021663 18491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:40.020870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.10.191.193:46179)
0620 23:11:40.021308 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:40.021454 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":238,"threads_started":1}
W20260620 23:11:40.022042 18140 heartbeater.cc:644] Failed to heartbeat to 127.10.191.254:39679 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
I20260620 23:11:40.023478 18492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:40.022639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.10.191.195:35353)
0620 23:11:40.023136 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:40.023291 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":264,"threads_started":1}
W20260620 23:11:40.023906 18423 heartbeater.cc:644] Failed to heartbeat to 127.10.191.254:39679 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
I20260620 23:11:40.534073 18493 raft_consensus.cc:482] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c8e2e42e0532441a97821bf64376c25d)
I20260620 23:11:40.534353 18493 raft_consensus.cc:504] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: false } } peers { permanent_uuid: "806cc6e3c2fa46b7a43e12917c49150c" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 44225 } attrs { promote: false } }
I20260620 23:11:40.534924 18494 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c8e2e42e0532441a97821bf64376c25d)
I20260620 23:11:40.535156 18494 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: false } } peers { permanent_uuid: "806cc6e3c2fa46b7a43e12917c49150c" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 44225 } attrs { promote: false } }
I20260620 23:11:40.535830 18493 leader_election.cc:288] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c8e2e42e0532441a97821bf64376c25d (127.10.191.254:39679), 6db99becd3324a5aaaf1eaf5985ae3f0 (127.10.191.253:33657)
I20260620 23:11:40.536597 18494 leader_election.cc:288] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c8e2e42e0532441a97821bf64376c25d (127.10.191.254:39679), 806cc6e3c2fa46b7a43e12917c49150c (127.10.191.252:44225)
I20260620 23:11:40.537302 18495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:40.536123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.0.0.1:38588)
0620 23:11:40.536630 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:40.536980 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":231,"threads_started":1}
W20260620 23:11:40.538455 17945 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
I20260620 23:11:40.539458 18496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:40.536720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.0.0.1:38602)
0620 23:11:40.538995 (+ 2275us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:40.539182 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2099,"thread_start_us":1946,"threads_started":1}
W20260620 23:11:40.541023 17748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
I20260620 23:11:40.542106 18497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:40.541375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.0.0.1:38616)
0620 23:11:40.541544 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:40.541866 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 23:11:40.542344 17945 leader_election.cc:334] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8e2e42e0532441a97821bf64376c25d (127.10.191.254:39679): Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
I20260620 23:11:40.545307 18502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:40.543861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.0.0.1:38618)
0620 23:11:40.544766 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:40.545004 (+ 238us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":788,"thread_start_us":660,"threads_started":1}
W20260620 23:11:40.545531 17748 leader_election.cc:334] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8e2e42e0532441a97821bf64376c25d (127.10.191.254:39679): Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
I20260620 23:11:40.546411 18495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:40.536769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:33657 (local address 127.0.0.1:42478)
0620 23:11:40.537625 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:40.537648 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:40.537838 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:40.539161 (+ 1323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:40.539166 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:40.539193 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:11:40.539484 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:40.539500 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:40.541513 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:40.541522 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:40.544155 (+ 2633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:40.544169 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:40.545600 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:40.545608 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:40.545803 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:40.545826 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 23:11:40.545940 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 23:11:40.546078 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"mutex_wait_us":23,"thread_start_us":1331,"threads_started":1}
I20260620 23:11:40.546445 18498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:40.536931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42478 (local address 127.10.191.253:33657)
0620 23:11:40.538783 (+ 1852us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:40.538791 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:40.538818 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:11:40.538905 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:40.538913 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:40.538997 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:40.539785 (+ 788us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:11:40.540061 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:40.541308 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:40.544310 (+ 3002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:40.545440 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:40.545631 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:40.545707 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:40.545961 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 23:11:40.546083 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1652,"thread_start_us":1447,"threads_started":1}
I20260620 23:11:40.547077 18496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:40.539976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:44225 (local address 127.0.0.1:38392)
0620 23:11:40.540112 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:40.540134 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:40.540261 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:40.540651 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:40.540656 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:40.540686 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:11:40.541011 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:40.541021 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:40.542702 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:40.542710 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:40.544814 (+ 2104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:40.544832 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:40.546273 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:40.546279 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:40.546549 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:40.546595 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 23:11:40.546679 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 23:11:40.546810 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 23:11:40.547227 18500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:40.539235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38392 (local address 127.10.191.252:44225)
0620 23:11:40.540095 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:40.540103 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:40.540361 (+ 258us) server_negotiation.cc:407] Connection header received
0620 23:11:40.540421 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:40.540426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:40.540493 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:40.540591 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:11:40.541181 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:40.542514 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:40.544985 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:40.546078 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:40.546273 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:40.546337 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:40.546728 (+ 391us) server_negotiation.cc:299] Negotiation successful
0620 23:11:40.546849 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":701,"thread_start_us":532,"threads_started":1}
I20260620 23:11:40.547149 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "806cc6e3c2fa46b7a43e12917c49150c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" is_pre_election: true
I20260620 23:11:40.547477 17771 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 806cc6e3c2fa46b7a43e12917c49150c in term 1.
I20260620 23:11:40.547791 17968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "806cc6e3c2fa46b7a43e12917c49150c" is_pre_election: true
I20260620 23:11:40.548027 17945 leader_election.cc:302] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6db99becd3324a5aaaf1eaf5985ae3f0, 806cc6e3c2fa46b7a43e12917c49150c; no voters: c8e2e42e0532441a97821bf64376c25d
I20260620 23:11:40.548158 17968 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6db99becd3324a5aaaf1eaf5985ae3f0 in term 1.
I20260620 23:11:40.548399 18493 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 23:11:40.548575 18493 raft_consensus.cc:482] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [term 1 FOLLOWER]: Starting leader election (detected failure of leader c8e2e42e0532441a97821bf64376c25d)
I20260620 23:11:40.548607 17748 leader_election.cc:302] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6db99becd3324a5aaaf1eaf5985ae3f0, 806cc6e3c2fa46b7a43e12917c49150c; no voters: c8e2e42e0532441a97821bf64376c25d
I20260620 23:11:40.548694 18493 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:11:40.548950 18494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 23:11:40.549091 18494 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c8e2e42e0532441a97821bf64376c25d)
I20260620 23:11:40.549182 18494 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:11:40.551903 18493 raft_consensus.cc:504] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: false } } peers { permanent_uuid: "806cc6e3c2fa46b7a43e12917c49150c" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 44225 } attrs { promote: false } }
I20260620 23:11:40.552170 18494 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: false } } peers { permanent_uuid: "806cc6e3c2fa46b7a43e12917c49150c" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 44225 } attrs { promote: false } }
I20260620 23:11:40.552625 18493 leader_election.cc:288] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [CANDIDATE]: Term 2 election: Requested vote from peers c8e2e42e0532441a97821bf64376c25d (127.10.191.254:39679), 6db99becd3324a5aaaf1eaf5985ae3f0 (127.10.191.253:33657)
I20260620 23:11:40.552820 18494 leader_election.cc:288] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [CANDIDATE]: Term 2 election: Requested vote from peers c8e2e42e0532441a97821bf64376c25d (127.10.191.254:39679), 806cc6e3c2fa46b7a43e12917c49150c (127.10.191.252:44225)
I20260620 23:11:40.553194 18495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:40.552755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.0.0.1:38628)
0620 23:11:40.552915 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:40.553018 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 23:11:40.553447 18496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:40.553050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.0.0.1:38642)
0620 23:11:40.553202 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:40.553290 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 23:11:40.553511 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "806cc6e3c2fa46b7a43e12917c49150c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0"
I20260620 23:11:40.553625 17968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "806cc6e3c2fa46b7a43e12917c49150c"
I20260620 23:11:40.553771 17771 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 806cc6e3c2fa46b7a43e12917c49150c in current term 2: Already voted for candidate 6db99becd3324a5aaaf1eaf5985ae3f0 in this term.
I20260620 23:11:40.553880 17968 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6db99becd3324a5aaaf1eaf5985ae3f0 in current term 2: Already voted for candidate 806cc6e3c2fa46b7a43e12917c49150c in this term.
I20260620 23:11:40.554692 18496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:40.554008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.0.0.1:38644)
0620 23:11:40.554479 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:40.554554 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357}
I20260620 23:11:40.554853 18495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:40.554481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.0.0.1:38652)
0620 23:11:40.554658 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:40.554721 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 23:11:40.554890 17748 leader_election.cc:334] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c8e2e42e0532441a97821bf64376c25d (127.10.191.254:39679): Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
I20260620 23:11:40.555028 17748 leader_election.cc:302] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6db99becd3324a5aaaf1eaf5985ae3f0; no voters: 806cc6e3c2fa46b7a43e12917c49150c, c8e2e42e0532441a97821bf64376c25d
W20260620 23:11:40.555043 17945 leader_election.cc:334] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c8e2e42e0532441a97821bf64376c25d (127.10.191.254:39679): Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
I20260620 23:11:40.555197 17945 leader_election.cc:302] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 806cc6e3c2fa46b7a43e12917c49150c; no voters: 6db99becd3324a5aaaf1eaf5985ae3f0, c8e2e42e0532441a97821bf64376c25d
I20260620 23:11:40.555284 18494 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 23:11:40.555626 18493 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 23:11:41.021574 18503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:41.020960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.10.191.194:48465)
0620 23:11:41.021345 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:41.021423 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":199,"threads_started":1}
I20260620 23:11:41.023509 18504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:41.022717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.10.191.193:32829)
0620 23:11:41.023224 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:41.023339 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":301,"threads_started":1}
I20260620 23:11:41.025226 18505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:41.024473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.10.191.195:46621)
0620 23:11:41.024884 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:41.024982 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":201,"threads_started":1}
I20260620 23:11:42.023175 18506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:42.022445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.10.191.194:52291)
0620 23:11:42.022911 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:42.023002 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":223,"threads_started":1}
I20260620 23:11:42.024941 18507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:42.024250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.10.191.193:56499)
0620 23:11:42.024658 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:42.024764 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":209,"threads_started":1}
I20260620 23:11:42.026608 18508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:42.025895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.10.191.195:57327)
0620 23:11:42.026356 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:42.026449 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":264,"threads_started":1}
I20260620 23:11:42.206674 18509 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:11:42.206923 18509 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: false } } peers { permanent_uuid: "806cc6e3c2fa46b7a43e12917c49150c" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 44225 } attrs { promote: false } }
I20260620 23:11:42.207577 18509 leader_election.cc:288] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c8e2e42e0532441a97821bf64376c25d (127.10.191.254:39679), 806cc6e3c2fa46b7a43e12917c49150c (127.10.191.252:44225)
I20260620 23:11:42.208572 18510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:42.207785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.0.0.1:38892)
0620 23:11:42.208233 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:42.208349 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":196,"threads_started":1}
I20260620 23:11:42.208726 17968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "806cc6e3c2fa46b7a43e12917c49150c" is_pre_election: true
I20260620 23:11:42.208954 17968 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6db99becd3324a5aaaf1eaf5985ae3f0 in term 2.
I20260620 23:11:42.209450 17748 leader_election.cc:302] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6db99becd3324a5aaaf1eaf5985ae3f0, 806cc6e3c2fa46b7a43e12917c49150c; no voters:
I20260620 23:11:42.209837 18509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 23:11:42.209985 18509 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:11:42.210095 18509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [term 2 FOLLOWER]: Advancing to term 3
I20260620 23:11:42.210450 18510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:42.210038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.0.0.1:38908)
0620 23:11:42.210230 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:42.210317 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 23:11:42.210676 17748 leader_election.cc:334] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c8e2e42e0532441a97821bf64376c25d (127.10.191.254:39679): Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
I20260620 23:11:42.212450 18509 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: false } } peers { permanent_uuid: "806cc6e3c2fa46b7a43e12917c49150c" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 44225 } attrs { promote: false } }
I20260620 23:11:42.212999 18509 leader_election.cc:288] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [CANDIDATE]: Term 3 election: Requested vote from peers c8e2e42e0532441a97821bf64376c25d (127.10.191.254:39679), 806cc6e3c2fa46b7a43e12917c49150c (127.10.191.252:44225)
I20260620 23:11:42.213461 18510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:42.213088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.0.0.1:38918)
0620 23:11:42.213242 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:42.213343 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 23:11:42.213661 17968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "806cc6e3c2fa46b7a43e12917c49150c"
I20260620 23:11:42.213886 17968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [term 2 FOLLOWER]: Advancing to term 3
I20260620 23:11:42.214463 18510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:42.214098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.0.0.1:38926)
0620 23:11:42.214245 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:42.214328 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 23:11:42.214658 17748 leader_election.cc:334] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer c8e2e42e0532441a97821bf64376c25d (127.10.191.254:39679): Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
I20260620 23:11:42.216336 17968 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6db99becd3324a5aaaf1eaf5985ae3f0 in term 3.
I20260620 23:11:42.216791 17748 leader_election.cc:302] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6db99becd3324a5aaaf1eaf5985ae3f0, 806cc6e3c2fa46b7a43e12917c49150c; no voters: c8e2e42e0532441a97821bf64376c25d
I20260620 23:11:42.217080 18509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 23:11:42.217438 18509 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [term 3 LEADER]: Becoming Leader. State: Replica: 6db99becd3324a5aaaf1eaf5985ae3f0, State: Running, Role: LEADER
I20260620 23:11:42.217815 18509 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: false } } peers { permanent_uuid: "806cc6e3c2fa46b7a43e12917c49150c" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 44225 } attrs { promote: false } }
I20260620 23:11:42.219455 18512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6db99becd3324a5aaaf1eaf5985ae3f0. Latest consensus state: current_term: 3 leader_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: false } } peers { permanent_uuid: "806cc6e3c2fa46b7a43e12917c49150c" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 44225 } attrs { promote: false } } }
I20260620 23:11:42.219731 18512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [sys.catalog]: This master's current role is: LEADER
I20260620 23:11:42.220345 18514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:11:42.223865 18514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:11:42.224602 18514 catalog_manager.cc:1259] Loaded cluster ID: c21dc8b8f31a4eff9c5507da7807c540
I20260620 23:11:42.224675 18514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:11:42.225708 18514 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:11:42.226452 18514 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0: Loaded TSK: 0
I20260620 23:11:42.227321 18514 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:11:42.246788 17760 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42476:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:11:42.254549 17760 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:11:42.258716 18510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:42.258336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.0.0.1:38942)
0620 23:11:42.258474 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:42.258555 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 23:11:42.259274 17968 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [term 3 FOLLOWER]: Refusing update from remote peer 6db99becd3324a5aaaf1eaf5985ae3f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260620 23:11:42.259946 18510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:42.259456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.0.0.1:38950)
0620 23:11:42.259704 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:42.259798 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 23:11:42.259994 18512 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "806cc6e3c2fa46b7a43e12917c49150c" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 44225 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260620 23:11:42.260298 17748 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 -> Peer c8e2e42e0532441a97821bf64376c25d (127.10.191.254:39679): Couldn't send request to peer c8e2e42e0532441a97821bf64376c25d. Status: Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 23:11:42.262687 18516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6db99becd3324a5aaaf1eaf5985ae3f0. Latest consensus state: current_term: 3 leader_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: false } } peers { permanent_uuid: "806cc6e3c2fa46b7a43e12917c49150c" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 44225 } attrs { promote: false } } }
I20260620 23:11:42.263023 18516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:11:42.265010 18509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: false } } peers { permanent_uuid: "806cc6e3c2fa46b7a43e12917c49150c" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 44225 } attrs { promote: false } } }
I20260620 23:11:42.265329 18509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0 [sys.catalog]: This master's current role is: LEADER
I20260620 23:11:42.271417 18516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c8e2e42e0532441a97821bf64376c25d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39679 } } peers { permanent_uuid: "6db99becd3324a5aaaf1eaf5985ae3f0" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33657 } attrs { promote: false } } peers { permanent_uuid: "806cc6e3c2fa46b7a43e12917c49150c" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 44225 } attrs { promote: false } } }
I20260620 23:11:42.271714 18516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 806cc6e3c2fa46b7a43e12917c49150c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:11:42.295898 18510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:42.275581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.194:35513 (local address 127.0.0.1:56504)
0620 23:11:42.275766 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:42.275784 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:42.275933 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:42.277437 (+ 1504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:42.277445 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:42.277483 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:11:42.277858 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:42.277874 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:42.279632 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:42.279641 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:42.291153 (+ 11512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:42.291176 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:42.294523 (+ 3347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:42.294552 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:42.294817 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:42.294975 (+ 158us) client_negotiation.cc:769] Sending connection context
0620 23:11:42.295055 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 23:11:42.295459 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":170}
I20260620 23:11:42.296890 18521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:42.275763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56504 (local address 127.10.191.194:35513)
0620 23:11:42.277067 (+ 1304us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:42.277074 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:42.277094 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:11:42.277187 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:42.277195 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:42.277273 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:42.277390 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:11:42.278042 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:42.279432 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:42.292068 (+ 12636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:42.294291 (+ 2223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:42.295351 (+ 1060us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:42.296321 (+ 970us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:42.296381 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 23:11:42.296524 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1145,"thread_start_us":1023,"threads_started":1}
I20260620 23:11:42.297797 18215 tablet_service.cc:1467] Processing CreateTablet for tablet 62a16726bf91431ea302c41f33843333 (DEFAULT_TABLE table=test-table [id=3cb2511981f24336a0461dccae0f28fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:11:42.298316 18215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62a16726bf91431ea302c41f33843333. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:11:42.300137 18525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:42.277271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53404 (local address 127.10.191.193:40211)
0620 23:11:42.279383 (+ 2112us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:42.279392 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:42.291791 (+ 12399us) server_negotiation.cc:407] Connection header received
0620 23:11:42.291960 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:42.291965 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:42.292076 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:42.292178 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:11:42.293010 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:42.294219 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:42.297454 (+ 3235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:42.298708 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:42.298900 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:42.298992 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:42.299603 (+ 611us) server_negotiation.cc:299] Negotiation successful
0620 23:11:42.299777 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1929,"thread_start_us":1791,"threads_started":1}
I20260620 23:11:42.301201 18523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:42.276375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50240 (local address 127.10.191.195:46823)
0620 23:11:42.280687 (+ 4312us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:42.280696 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:42.289077 (+ 8381us) server_negotiation.cc:407] Connection header received
0620 23:11:42.289424 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:42.289429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:42.289493 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:42.289865 (+ 372us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:11:42.293193 (+ 3328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:42.294645 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:42.297782 (+ 3137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:42.299206 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:42.299389 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:42.299473 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:42.300694 (+ 1221us) server_negotiation.cc:299] Negotiation successful
0620 23:11:42.300868 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4139,"thread_start_us":3975,"threads_started":1}
I20260620 23:11:42.300534 18522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:42.276180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.195:46823 (local address 127.0.0.1:50240)
0620 23:11:42.288769 (+ 12589us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:42.288786 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:42.288953 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:42.289874 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:42.289878 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:42.289901 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:11:42.290304 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:42.290318 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:42.294823 (+ 4505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:42.294833 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:42.297558 (+ 2725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:42.297577 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:42.299635 (+ 2058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:42.299642 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:42.299833 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:42.299869 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 23:11:42.299953 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 23:11:42.300097 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12425,"mutex_wait_us":94,"thread_start_us":12286,"threads_started":1}
I20260620 23:11:42.301913 18524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:42.276840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:40211 (local address 127.0.0.1:53404)
0620 23:11:42.291640 (+ 14800us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:42.291656 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:42.291834 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:42.292399 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:42.292404 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:42.292429 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:11:42.292813 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:42.292826 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:42.294407 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:42.294416 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:42.297256 (+ 2840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:42.297272 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:42.299189 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:42.299195 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:42.299400 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:42.299439 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 23:11:42.300399 (+ 960us) client_negotiation.cc:240] Negotiation successful
0620 23:11:42.300553 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14668,"mutex_wait_us":149,"thread_start_us":14517,"threads_started":1}
I20260620 23:11:42.302289 18357 tablet_service.cc:1467] Processing CreateTablet for tablet 62a16726bf91431ea302c41f33843333 (DEFAULT_TABLE table=test-table [id=3cb2511981f24336a0461dccae0f28fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:11:42.302789 18357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62a16726bf91431ea302c41f33843333. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:11:42.303016 18074 tablet_service.cc:1467] Processing CreateTablet for tablet 62a16726bf91431ea302c41f33843333 (DEFAULT_TABLE table=test-table [id=3cb2511981f24336a0461dccae0f28fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:11:42.303470 18074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62a16726bf91431ea302c41f33843333. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:11:42.307552 18526 tablet_bootstrap.cc:492] T 62a16726bf91431ea302c41f33843333 P ab48ee8fa5634fc59ab80e8a1c3d5361: Bootstrap starting.
I20260620 23:11:42.309360 18526 tablet_bootstrap.cc:654] T 62a16726bf91431ea302c41f33843333 P ab48ee8fa5634fc59ab80e8a1c3d5361: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:42.311475 18528 tablet_bootstrap.cc:492] T 62a16726bf91431ea302c41f33843333 P f3868efeb1fa4926a2fa269532dddbab: Bootstrap starting.
I20260620 23:11:42.312911 18528 tablet_bootstrap.cc:654] T 62a16726bf91431ea302c41f33843333 P f3868efeb1fa4926a2fa269532dddbab: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:42.314179 18527 tablet_bootstrap.cc:492] T 62a16726bf91431ea302c41f33843333 P 25499f9be0e240fbbcf6d2864c0b51bf: Bootstrap starting.
I20260620 23:11:42.316464 18527 tablet_bootstrap.cc:654] T 62a16726bf91431ea302c41f33843333 P 25499f9be0e240fbbcf6d2864c0b51bf: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:42.318846 18528 tablet_bootstrap.cc:492] T 62a16726bf91431ea302c41f33843333 P f3868efeb1fa4926a2fa269532dddbab: No bootstrap required, opened a new log
I20260620 23:11:42.319031 18528 ts_tablet_manager.cc:1397] T 62a16726bf91431ea302c41f33843333 P f3868efeb1fa4926a2fa269532dddbab: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260620 23:11:42.320024 18528 raft_consensus.cc:348] T 62a16726bf91431ea302c41f33843333 P f3868efeb1fa4926a2fa269532dddbab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab48ee8fa5634fc59ab80e8a1c3d5361" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 35513 } } peers { permanent_uuid: "25499f9be0e240fbbcf6d2864c0b51bf" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46823 } } peers { permanent_uuid: "f3868efeb1fa4926a2fa269532dddbab" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40211 } }
I20260620 23:11:42.320294 18528 raft_consensus.cc:374] T 62a16726bf91431ea302c41f33843333 P f3868efeb1fa4926a2fa269532dddbab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:42.320410 18528 raft_consensus.cc:729] T 62a16726bf91431ea302c41f33843333 P f3868efeb1fa4926a2fa269532dddbab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3868efeb1fa4926a2fa269532dddbab, State: Initialized, Role: FOLLOWER
I20260620 23:11:42.321012 18528 consensus_queue.cc:260] T 62a16726bf91431ea302c41f33843333 P f3868efeb1fa4926a2fa269532dddbab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab48ee8fa5634fc59ab80e8a1c3d5361" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 35513 } } peers { permanent_uuid: "25499f9be0e240fbbcf6d2864c0b51bf" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46823 } } peers { permanent_uuid: "f3868efeb1fa4926a2fa269532dddbab" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40211 } }
I20260620 23:11:42.322080 18528 ts_tablet_manager.cc:1428] T 62a16726bf91431ea302c41f33843333 P f3868efeb1fa4926a2fa269532dddbab: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 23:11:42.322952 18507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:42.322281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.10.191.193:45493)
0620 23:11:42.322491 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:42.322636 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 23:11:42.322978 18532 raft_consensus.cc:482] T 62a16726bf91431ea302c41f33843333 P f3868efeb1fa4926a2fa269532dddbab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:11:42.323365 18532 raft_consensus.cc:504] T 62a16726bf91431ea302c41f33843333 P f3868efeb1fa4926a2fa269532dddbab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab48ee8fa5634fc59ab80e8a1c3d5361" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 35513 } } peers { permanent_uuid: "25499f9be0e240fbbcf6d2864c0b51bf" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46823 } } peers { permanent_uuid: "f3868efeb1fa4926a2fa269532dddbab" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40211 } }
I20260620 23:11:42.324319 18507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:42.323742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.10.191.193:40037)
0620 23:11:42.324056 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:42.324153 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209}
I20260620 23:11:42.325350 18141 heartbeater.cc:503] Master 127.10.191.253:33657 was elected leader, sending a full tablet report...
I20260620 23:11:42.330235 18532 leader_election.cc:288] T 62a16726bf91431ea302c41f33843333 P f3868efeb1fa4926a2fa269532dddbab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab48ee8fa5634fc59ab80e8a1c3d5361 (127.10.191.194:35513), 25499f9be0e240fbbcf6d2864c0b51bf (127.10.191.195:46823)
I20260620 23:11:42.334647 18521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:42.326642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:53923 (local address 127.10.191.194:35513)
0620 23:11:42.326846 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:42.326852 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:42.327312 (+ 460us) server_negotiation.cc:407] Connection header received
0620 23:11:42.327395 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:42.327401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:42.327482 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:42.327593 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:11:42.328150 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:42.329030 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:42.331931 (+ 2901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:42.333338 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:42.333478 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:42.333595 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:42.334240 (+ 645us) server_negotiation.cc:299] Negotiation successful
0620 23:11:42.334386 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 23:11:42.335359 18507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:42.326307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.194:35513 (local address 127.10.191.193:53923)
0620 23:11:42.327025 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:42.327047 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:42.327174 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:42.327615 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:42.327620 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:42.327646 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:11:42.327992 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:42.328009 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:42.329181 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:42.329187 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:42.331745 (+ 2558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:42.331759 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:42.333802 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:42.333809 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:42.334026 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:42.334063 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 23:11:42.334832 (+ 769us) client_negotiation.cc:240] Negotiation successful
0620 23:11:42.335025 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601}
I20260620 23:11:42.335987 18235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a16726bf91431ea302c41f33843333" candidate_uuid: "f3868efeb1fa4926a2fa269532dddbab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab48ee8fa5634fc59ab80e8a1c3d5361" is_pre_election: true
W20260620 23:11:42.336967 18030 leader_election.cc:341] T 62a16726bf91431ea302c41f33843333 P f3868efeb1fa4926a2fa269532dddbab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab48ee8fa5634fc59ab80e8a1c3d5361 (127.10.191.194:35513): Illegal state: must be running to vote when last-logged opid is not known
I20260620 23:11:42.338013 18523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:42.330034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:47083 (local address 127.10.191.195:46823)
0620 23:11:42.330231 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:42.330238 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:42.330389 (+ 151us) server_negotiation.cc:407] Connection header received
0620 23:11:42.330574 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:42.330581 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:42.330674 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:42.330773 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:11:42.331475 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:42.332610 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:42.335734 (+ 3124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:42.336974 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:42.337183 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:42.337283 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:42.337700 (+ 417us) server_negotiation.cc:299] Negotiation successful
0620 23:11:42.337835 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 23:11:42.338815 18533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:42.329593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.195:46823 (local address 127.10.191.193:47083)
0620 23:11:42.330309 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:42.330328 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:42.330448 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:42.330818 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:42.330823 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:42.330850 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:11:42.331285 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:42.331298 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:42.332811 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:42.332818 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:42.335307 (+ 2489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:42.335321 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:42.337297 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:42.337307 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:42.337483 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:42.337517 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 23:11:42.338219 (+ 702us) client_negotiation.cc:240] Negotiation successful
0620 23:11:42.338397 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":295,"threads_started":1}
I20260620 23:11:42.339428 18526 tablet_bootstrap.cc:492] T 62a16726bf91431ea302c41f33843333 P ab48ee8fa5634fc59ab80e8a1c3d5361: No bootstrap required, opened a new log
I20260620 23:11:42.339594 18526 ts_tablet_manager.cc:1397] T 62a16726bf91431ea302c41f33843333 P ab48ee8fa5634fc59ab80e8a1c3d5361: Time spent bootstrapping tablet: real 0.032s user 0.006s sys 0.007s
I20260620 23:11:42.339656 18377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a16726bf91431ea302c41f33843333" candidate_uuid: "f3868efeb1fa4926a2fa269532dddbab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25499f9be0e240fbbcf6d2864c0b51bf" is_pre_election: true
I20260620 23:11:42.340515 18527 tablet_bootstrap.cc:492] T 62a16726bf91431ea302c41f33843333 P 25499f9be0e240fbbcf6d2864c0b51bf: No bootstrap required, opened a new log
I20260620 23:11:42.340431 18526 raft_consensus.cc:348] T 62a16726bf91431ea302c41f33843333 P ab48ee8fa5634fc59ab80e8a1c3d5361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab48ee8fa5634fc59ab80e8a1c3d5361" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 35513 } } peers { permanent_uuid: "25499f9be0e240fbbcf6d2864c0b51bf" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46823 } } peers { permanent_uuid: "f3868efeb1fa4926a2fa269532dddbab" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40211 } }
I20260620 23:11:42.340657 18527 ts_tablet_manager.cc:1397] T 62a16726bf91431ea302c41f33843333 P 25499f9be0e240fbbcf6d2864c0b51bf: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s
W20260620 23:11:42.340628 18027 leader_election.cc:341] T 62a16726bf91431ea302c41f33843333 P f3868efeb1fa4926a2fa269532dddbab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 25499f9be0e240fbbcf6d2864c0b51bf (127.10.191.195:46823): Illegal state: must be running to vote when last-logged opid is not known
I20260620 23:11:42.340698 18526 raft_consensus.cc:374] T 62a16726bf91431ea302c41f33843333 P ab48ee8fa5634fc59ab80e8a1c3d5361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:42.340793 18526 raft_consensus.cc:729] T 62a16726bf91431ea302c41f33843333 P ab48ee8fa5634fc59ab80e8a1c3d5361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab48ee8fa5634fc59ab80e8a1c3d5361, State: Initialized, Role: FOLLOWER
I20260620 23:11:42.340767 18027 leader_election.cc:302] T 62a16726bf91431ea302c41f33843333 P f3868efeb1fa4926a2fa269532dddbab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f3868efeb1fa4926a2fa269532dddbab; no voters: 25499f9be0e240fbbcf6d2864c0b51bf, ab48ee8fa5634fc59ab80e8a1c3d5361
I20260620 23:11:42.341055 18526 consensus_queue.cc:260] T 62a16726bf91431ea302c41f33843333 P ab48ee8fa5634fc59ab80e8a1c3d5361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab48ee8fa5634fc59ab80e8a1c3d5361" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 35513 } } peers { permanent_uuid: "25499f9be0e240fbbcf6d2864c0b51bf" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46823 } } peers { permanent_uuid: "f3868efeb1fa4926a2fa269532dddbab" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40211 } }
I20260620 23:11:42.341202 18532 raft_consensus.cc:2726] T 62a16726bf91431ea302c41f33843333 P f3868efeb1fa4926a2fa269532dddbab [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 23:11:42.341452 18527 raft_consensus.cc:348] T 62a16726bf91431ea302c41f33843333 P 25499f9be0e240fbbcf6d2864c0b51bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab48ee8fa5634fc59ab80e8a1c3d5361" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 35513 } } peers { permanent_uuid: "25499f9be0e240fbbcf6d2864c0b51bf" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46823 } } peers { permanent_uuid: "f3868efeb1fa4926a2fa269532dddbab" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40211 } }
I20260620 23:11:42.341668 18527 raft_consensus.cc:374] T 62a16726bf91431ea302c41f33843333 P 25499f9be0e240fbbcf6d2864c0b51bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:42.341751 18527 raft_consensus.cc:729] T 62a16726bf91431ea302c41f33843333 P 25499f9be0e240fbbcf6d2864c0b51bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25499f9be0e240fbbcf6d2864c0b51bf, State: Initialized, Role: FOLLOWER
I20260620 23:11:42.341958 18527 consensus_queue.cc:260] T 62a16726bf91431ea302c41f33843333 P 25499f9be0e240fbbcf6d2864c0b51bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab48ee8fa5634fc59ab80e8a1c3d5361" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 35513 } } peers { permanent_uuid: "25499f9be0e240fbbcf6d2864c0b51bf" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46823 } } peers { permanent_uuid: "f3868efeb1fa4926a2fa269532dddbab" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40211 } }
I20260620 23:11:42.342742 18527 ts_tablet_manager.cc:1428] T 62a16726bf91431ea302c41f33843333 P 25499f9be0e240fbbcf6d2864c0b51bf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:11:42.343564 18508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:42.343186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.10.191.195:38279)
0620 23:11:42.343332 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:42.343427 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 23:11:42.344035 18526 ts_tablet_manager.cc:1428] T 62a16726bf91431ea302c41f33843333 P ab48ee8fa5634fc59ab80e8a1c3d5361: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260620 23:11:42.345412 18506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:42.344974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.10.191.194:60939)
0620 23:11:42.345157 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:42.345259 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 23:11:42.346679 18508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:42.346139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.10.191.195:33029)
0620 23:11:42.346418 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:42.346513 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260620 23:11:42.347640 18506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:42.347196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.10.191.194:52825)
0620 23:11:42.347374 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:42.347469 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 23:11:42.348762 18282 heartbeater.cc:503] Master 127.10.191.253:33657 was elected leader, sending a full tablet report...
W20260620 23:11:42.371320 18283 tablet.cc:2369] T 62a16726bf91431ea302c41f33843333 P ab48ee8fa5634fc59ab80e8a1c3d5361: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:11:42.423084 18535 raft_consensus.cc:482] T 62a16726bf91431ea302c41f33843333 P 25499f9be0e240fbbcf6d2864c0b51bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:11:42.423285 18535 raft_consensus.cc:504] T 62a16726bf91431ea302c41f33843333 P 25499f9be0e240fbbcf6d2864c0b51bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab48ee8fa5634fc59ab80e8a1c3d5361" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 35513 } } peers { permanent_uuid: "25499f9be0e240fbbcf6d2864c0b51bf" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46823 } } peers { permanent_uuid: "f3868efeb1fa4926a2fa269532dddbab" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40211 } }
I20260620 23:11:42.424139 18535 leader_election.cc:288] T 62a16726bf91431ea302c41f33843333 P 25499f9be0e240fbbcf6d2864c0b51bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab48ee8fa5634fc59ab80e8a1c3d5361 (127.10.191.194:35513), f3868efeb1fa4926a2fa269532dddbab (127.10.191.193:40211)
I20260620 23:11:42.424736 18235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a16726bf91431ea302c41f33843333" candidate_uuid: "25499f9be0e240fbbcf6d2864c0b51bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab48ee8fa5634fc59ab80e8a1c3d5361" is_pre_election: true
I20260620 23:11:42.424750 18093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a16726bf91431ea302c41f33843333" candidate_uuid: "25499f9be0e240fbbcf6d2864c0b51bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3868efeb1fa4926a2fa269532dddbab" is_pre_election: true
I20260620 23:11:42.425014 18235 raft_consensus.cc:2454] T 62a16726bf91431ea302c41f33843333 P ab48ee8fa5634fc59ab80e8a1c3d5361 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25499f9be0e240fbbcf6d2864c0b51bf in term 0.
I20260620 23:11:42.425035 18093 raft_consensus.cc:2454] T 62a16726bf91431ea302c41f33843333 P f3868efeb1fa4926a2fa269532dddbab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25499f9be0e240fbbcf6d2864c0b51bf in term 0.
I20260620 23:11:42.425484 18313 leader_election.cc:302] T 62a16726bf91431ea302c41f33843333 P 25499f9be0e240fbbcf6d2864c0b51bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 25499f9be0e240fbbcf6d2864c0b51bf, ab48ee8fa5634fc59ab80e8a1c3d5361, f3868efeb1fa4926a2fa269532dddbab; no voters:
I20260620 23:11:42.425760 18535 raft_consensus.cc:2781] T 62a16726bf91431ea302c41f33843333 P 25499f9be0e240fbbcf6d2864c0b51bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:11:42.425889 18535 raft_consensus.cc:482] T 62a16726bf91431ea302c41f33843333 P 25499f9be0e240fbbcf6d2864c0b51bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:11:42.425999 18535 raft_consensus.cc:3037] T 62a16726bf91431ea302c41f33843333 P 25499f9be0e240fbbcf6d2864c0b51bf [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:42.428020 18535 raft_consensus.cc:504] T 62a16726bf91431ea302c41f33843333 P 25499f9be0e240fbbcf6d2864c0b51bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab48ee8fa5634fc59ab80e8a1c3d5361" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 35513 } } peers { permanent_uuid: "25499f9be0e240fbbcf6d2864c0b51bf" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46823 } } peers { permanent_uuid: "f3868efeb1fa4926a2fa269532dddbab" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40211 } }
I20260620 23:11:42.428529 18535 leader_election.cc:288] T 62a16726bf91431ea302c41f33843333 P 25499f9be0e240fbbcf6d2864c0b51bf [CANDIDATE]: Term 1 election: Requested vote from peers ab48ee8fa5634fc59ab80e8a1c3d5361 (127.10.191.194:35513), f3868efeb1fa4926a2fa269532dddbab (127.10.191.193:40211)
I20260620 23:11:42.429049 18235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a16726bf91431ea302c41f33843333" candidate_uuid: "25499f9be0e240fbbcf6d2864c0b51bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab48ee8fa5634fc59ab80e8a1c3d5361"
I20260620 23:11:42.429069 18093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a16726bf91431ea302c41f33843333" candidate_uuid: "25499f9be0e240fbbcf6d2864c0b51bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3868efeb1fa4926a2fa269532dddbab"
I20260620 23:11:42.429253 18093 raft_consensus.cc:3037] T 62a16726bf91431ea302c41f33843333 P f3868efeb1fa4926a2fa269532dddbab [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:42.429275 18235 raft_consensus.cc:3037] T 62a16726bf91431ea302c41f33843333 P ab48ee8fa5634fc59ab80e8a1c3d5361 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:42.432192 18235 raft_consensus.cc:2454] T 62a16726bf91431ea302c41f33843333 P ab48ee8fa5634fc59ab80e8a1c3d5361 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25499f9be0e240fbbcf6d2864c0b51bf in term 1.
I20260620 23:11:42.432220 18093 raft_consensus.cc:2454] T 62a16726bf91431ea302c41f33843333 P f3868efeb1fa4926a2fa269532dddbab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25499f9be0e240fbbcf6d2864c0b51bf in term 1.
I20260620 23:11:42.432621 18313 leader_election.cc:302] T 62a16726bf91431ea302c41f33843333 P 25499f9be0e240fbbcf6d2864c0b51bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 25499f9be0e240fbbcf6d2864c0b51bf, ab48ee8fa5634fc59ab80e8a1c3d5361, f3868efeb1fa4926a2fa269532dddbab; no voters:
I20260620 23:11:42.432873 18535 raft_consensus.cc:2781] T 62a16726bf91431ea302c41f33843333 P 25499f9be0e240fbbcf6d2864c0b51bf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:11:42.433277 18535 raft_consensus.cc:686] T 62a16726bf91431ea302c41f33843333 P 25499f9be0e240fbbcf6d2864c0b51bf [term 1 LEADER]: Becoming Leader. State: Replica: 25499f9be0e240fbbcf6d2864c0b51bf, State: Running, Role: LEADER
I20260620 23:11:42.433624 18424 heartbeater.cc:503] Master 127.10.191.253:33657 was elected leader, sending a full tablet report...
I20260620 23:11:42.433600 18535 consensus_queue.cc:237] T 62a16726bf91431ea302c41f33843333 P 25499f9be0e240fbbcf6d2864c0b51bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab48ee8fa5634fc59ab80e8a1c3d5361" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 35513 } } peers { permanent_uuid: "25499f9be0e240fbbcf6d2864c0b51bf" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46823 } } peers { permanent_uuid: "f3868efeb1fa4926a2fa269532dddbab" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40211 } }
I20260620 23:11:42.434392 18508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:42.433934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.10.191.195:41839)
0620 23:11:42.434116 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:42.434230 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 23:11:42.436414 17761 catalog_manager.cc:5495] T 62a16726bf91431ea302c41f33843333 P 25499f9be0e240fbbcf6d2864c0b51bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 25499f9be0e240fbbcf6d2864c0b51bf (127.10.191.195). New cstate: current_term: 1 leader_uuid: "25499f9be0e240fbbcf6d2864c0b51bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab48ee8fa5634fc59ab80e8a1c3d5361" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 35513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25499f9be0e240fbbcf6d2864c0b51bf" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3868efeb1fa4926a2fa269532dddbab" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40211 } health_report { overall_health: UNKNOWN } } }
I20260620 23:11:42.488380 18523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:42.483876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50250 (local address 127.10.191.195:46823)
0620 23:11:42.484062 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:42.484069 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:42.484091 (+ 22us) server_negotiation.cc:407] Connection header received
0620 23:11:42.484171 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:42.484175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:42.484239 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:42.484337 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:11:42.484705 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:42.485463 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:42.486735 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:42.487145 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:42.487283 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:42.487669 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:11:42.487773 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:11:42.487871 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:42.488014 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 23:11:42.488089 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 23:11:42.491386 18235 raft_consensus.cc:1261] T 62a16726bf91431ea302c41f33843333 P ab48ee8fa5634fc59ab80e8a1c3d5361 [term 1 FOLLOWER]: Refusing update from remote peer 25499f9be0e240fbbcf6d2864c0b51bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:11:42.491441 18093 raft_consensus.cc:1261] T 62a16726bf91431ea302c41f33843333 P f3868efeb1fa4926a2fa269532dddbab [term 1 FOLLOWER]: Refusing update from remote peer 25499f9be0e240fbbcf6d2864c0b51bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:11:42.492077 18536 consensus_queue.cc:1035] T 62a16726bf91431ea302c41f33843333 P 25499f9be0e240fbbcf6d2864c0b51bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab48ee8fa5634fc59ab80e8a1c3d5361" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 35513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:11:42.492523 18535 consensus_queue.cc:1035] T 62a16726bf91431ea302c41f33843333 P 25499f9be0e240fbbcf6d2864c0b51bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3868efeb1fa4926a2fa269532dddbab" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:11:42.495316 18506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:42.494619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.10.191.194:55993)
0620 23:11:42.494974 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:42.495109 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260620 23:11:42.496088 18533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:42.495690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.10.191.193:57923)
0620 23:11:42.495818 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:42.495921 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 23:11:42.507975 18508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:42.507253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.10.191.195:35623)
0620 23:11:42.507618 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:42.507730 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224}
I20260620 23:11:42.508350 18506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:42.507641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.10.191.194:39775)
0620 23:11:42.507989 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:42.508134 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197}
I20260620 23:11:42.510133 17760 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:11:42.510486 17760 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:11:42.516664 18533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:42.516227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.10.191.193:32883)
0620 23:11:42.516394 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:42.516514 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 23:11:42.517387 18508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:42.516691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.10.191.195:37829)
0620 23:11:42.517076 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:42.517187 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229}
I20260620 23:11:42.518349 17760 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6db99becd3324a5aaaf1eaf5985ae3f0: Sending DeleteTablet for 3 replicas of tablet 62a16726bf91431ea302c41f33843333
I20260620 23:11:42.519650 18357 tablet_service.cc:1514] Processing DeleteTablet for tablet 62a16726bf91431ea302c41f33843333 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:11:42 UTC) from {username='oryx', principal='oryx/127.10.191.253@KRBTEST.COM'} at 127.0.0.1:50240
I20260620 23:11:42.519884 18215 tablet_service.cc:1514] Processing DeleteTablet for tablet 62a16726bf91431ea302c41f33843333 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:11:42 UTC) from {username='oryx', principal='oryx/127.10.191.253@KRBTEST.COM'} at 127.0.0.1:56504
I20260620 23:11:42.520026 18074 tablet_service.cc:1514] Processing DeleteTablet for tablet 62a16726bf91431ea302c41f33843333 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:11:42 UTC) from {username='oryx', principal='oryx/127.10.191.253@KRBTEST.COM'} at 127.0.0.1:53404
I20260620 23:11:42.524868 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 17941
I20260620 23:11:42.532243 18551 tablet_replica.cc:330] T 62a16726bf91431ea302c41f33843333 P ab48ee8fa5634fc59ab80e8a1c3d5361: stopping tablet replica
I20260620 23:11:42.532459 18551 raft_consensus.cc:2229] T 62a16726bf91431ea302c41f33843333 P ab48ee8fa5634fc59ab80e8a1c3d5361 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:11:42.532735 18551 raft_consensus.cc:2258] T 62a16726bf91431ea302c41f33843333 P ab48ee8fa5634fc59ab80e8a1c3d5361 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:11:42.533932 18552 tablet_replica.cc:330] T 62a16726bf91431ea302c41f33843333 P f3868efeb1fa4926a2fa269532dddbab: stopping tablet replica
I20260620 23:11:42.534114 18551 ts_tablet_manager.cc:1905] T 62a16726bf91431ea302c41f33843333 P ab48ee8fa5634fc59ab80e8a1c3d5361: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:11:42.539847 18552 raft_consensus.cc:2229] T 62a16726bf91431ea302c41f33843333 P f3868efeb1fa4926a2fa269532dddbab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:11:42.540568 18552 raft_consensus.cc:2258] T 62a16726bf91431ea302c41f33843333 P f3868efeb1fa4926a2fa269532dddbab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:11:42.542026 18551 ts_tablet_manager.cc:1918] T 62a16726bf91431ea302c41f33843333 P ab48ee8fa5634fc59ab80e8a1c3d5361: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:11:42.542163 18551 log.cc:1196] T 62a16726bf91431ea302c41f33843333 P ab48ee8fa5634fc59ab80e8a1c3d5361: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-1/wal/wals/62a16726bf91431ea302c41f33843333
I20260620 23:11:42.542332 18552 ts_tablet_manager.cc:1905] T 62a16726bf91431ea302c41f33843333 P f3868efeb1fa4926a2fa269532dddbab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:11:42.542647 18551 ts_tablet_manager.cc:1939] T 62a16726bf91431ea302c41f33843333 P ab48ee8fa5634fc59ab80e8a1c3d5361: Deleting consensus metadata
I20260620 23:11:42.543372 18550 tablet_replica.cc:330] T 62a16726bf91431ea302c41f33843333 P 25499f9be0e240fbbcf6d2864c0b51bf: stopping tablet replica
I20260620 23:11:42.543556 18550 raft_consensus.cc:2229] T 62a16726bf91431ea302c41f33843333 P 25499f9be0e240fbbcf6d2864c0b51bf [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:11:42.543903 17748 catalog_manager.cc:4841] TS ab48ee8fa5634fc59ab80e8a1c3d5361 (127.10.191.194:35513): tablet 62a16726bf91431ea302c41f33843333 (table test-table [id=3cb2511981f24336a0461dccae0f28fc]) successfully deleted
I20260620 23:11:42.544070 18550 raft_consensus.cc:2258] T 62a16726bf91431ea302c41f33843333 P 25499f9be0e240fbbcf6d2864c0b51bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:11:42.545010 18550 ts_tablet_manager.cc:1905] T 62a16726bf91431ea302c41f33843333 P 25499f9be0e240fbbcf6d2864c0b51bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:11:42.548575 18552 ts_tablet_manager.cc:1918] T 62a16726bf91431ea302c41f33843333 P f3868efeb1fa4926a2fa269532dddbab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:11:42.548739 18552 log.cc:1196] T 62a16726bf91431ea302c41f33843333 P f3868efeb1fa4926a2fa269532dddbab: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-0/wal/wals/62a16726bf91431ea302c41f33843333
I20260620 23:11:42.549784 18552 ts_tablet_manager.cc:1939] T 62a16726bf91431ea302c41f33843333 P f3868efeb1fa4926a2fa269532dddbab: Deleting consensus metadata
I20260620 23:11:42.551002 17746 catalog_manager.cc:4841] TS f3868efeb1fa4926a2fa269532dddbab (127.10.191.193:40211): tablet 62a16726bf91431ea302c41f33843333 (table test-table [id=3cb2511981f24336a0461dccae0f28fc]) successfully deleted
I20260620 23:11:42.551573 18550 ts_tablet_manager.cc:1918] T 62a16726bf91431ea302c41f33843333 P 25499f9be0e240fbbcf6d2864c0b51bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:11:42.551751 18550 log.cc:1196] T 62a16726bf91431ea302c41f33843333 P 25499f9be0e240fbbcf6d2864c0b51bf: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781997042893763-11007-0/minicluster-data/ts-2/wal/wals/62a16726bf91431ea302c41f33843333
I20260620 23:11:42.552186 18550 ts_tablet_manager.cc:1939] T 62a16726bf91431ea302c41f33843333 P 25499f9be0e240fbbcf6d2864c0b51bf: Deleting consensus metadata
I20260620 23:11:42.553223 17745 catalog_manager.cc:4841] TS 25499f9be0e240fbbcf6d2864c0b51bf (127.10.191.195:46823): tablet 62a16726bf91431ea302c41f33843333 (table test-table [id=3cb2511981f24336a0461dccae0f28fc]) successfully deleted
I20260620 23:11:42.558421 18525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:42.525942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53414 (local address 127.10.191.193:40211)
0620 23:11:42.526136 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:42.526143 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:42.539577 (+ 13434us) server_negotiation.cc:407] Connection header received
0620 23:11:42.539744 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:42.539748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:42.539823 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:42.539939 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:42.545675 (+ 5736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:42.546794 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:42.548609 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:42.549300 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:42.549410 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:42.552659 (+ 3249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:42.552671 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:42.552687 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:42.552722 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:42.555752 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:42.556415 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:42.556422 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:42.556424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:42.556554 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:42.556944 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:42.556951 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:42.556959 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:42.557302 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:42.557589 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:42.557880 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 23:11:42.558116 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 23:11:42.559073 18134 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:11:42.652750 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 17941
I20260620 23:11:42.670679 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 18151
I20260620 23:11:42.674192 18524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:42.673783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.0.0.1:38952)
0620 23:11:42.673952 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:42.674050 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 23:11:42.675191 18524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:42.674833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39679 (local address 127.0.0.1:38966)
0620 23:11:42.674995 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:42.675062 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 23:11:42.684922 18521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:42.671468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56508 (local address 127.10.191.194:35513)
0620 23:11:42.671699 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:42.671706 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:42.671822 (+ 116us) server_negotiation.cc:407] Connection header received
0620 23:11:42.672124 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:42.672129 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:42.672196 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:42.672280 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:42.673796 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:42.674725 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:42.676138 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:42.676530 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:42.676644 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:42.679301 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:42.679316 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:42.679335 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:42.679372 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:42.682194 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:42.683065 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:42.683077 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:42.683084 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:42.683249 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:42.683608 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:42.683612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:42.683614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:42.683970 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:42.684197 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:42.684473 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 23:11:42.684638 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 23:11:42.685623 18275 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:11:42.792994 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 18151
I20260620 23:11:42.810305 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 18293
I20260620 23:11:42.823233 18523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:42.811130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50266 (local address 127.10.191.195:46823)
0620 23:11:42.811341 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:42.811346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:42.811365 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:11:42.811563 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:42.811568 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:42.811644 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:42.811743 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:42.813032 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:42.813736 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:42.814857 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:42.815459 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:42.815554 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:42.818131 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:42.818143 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:42.818159 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:42.818190 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:42.820768 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:42.821375 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:42.821382 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:42.821384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:42.821501 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:42.821858 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:42.821861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:42.821863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:42.822235 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:42.822450 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:42.822694 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 23:11:42.822925 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 23:11:42.823944 18417 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:11:42.941310 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 18293
I20260620 23:11:42.963340 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 17619
I20260620 23:11:42.964396 17791 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:11:43.085489 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 17619
I20260620 23:11:43.106652 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 17719
I20260620 23:11:43.107765 17988 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:11:43.221118 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 17719
2026-06-20T23:11:43Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11096 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:11:43 dist-test-slave-jkbq krb5kdc[18561](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:11:43 dist-test-slave-jkbq krb5kdc[18561](info): set up 2 sockets
Jun 20 23:11:43 dist-test-slave-jkbq krb5kdc[18561](info): commencing operation
krb5kdc: starting...
W20260620 23:11:44.323936 11007 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Jun 20 23:11:44 dist-test-slave-jkbq krb5kdc[18561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:11:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:11:44Z Disabled control of system clock
WARNING: no policy specified for oryx/127.10.191.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.191.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.191.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.254.keytab.
Entry for principal oryx/127.10.191.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.254.keytab.
I20260620 23:11:44.534008 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:36897
--webserver_interface=127.10.191.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.254.keytab
--principal=oryx/127.10.191.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:42351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:36897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:11:44.847846 18574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:44.848169 18574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:44.848351 18574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:44.855381 18574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:11:44.855518 18574 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 23:11:44.855618 18574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:44.855746 18574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:44.855831 18574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:11:44.855919 18574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:11:44.865818 18574 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:42351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:36897
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.254.keytab
--principal=oryx/127.10.191.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:36897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:44.867139 18574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:44.868455 18574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:44.880734 18582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:44.880775 18579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:44.880734 18574 server_base.cc:1034] running on GCE node
W20260620 23:11:44.880728 18580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:44.881904 18574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:44.883297 18574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:44.884564 18574 hybrid_clock.cc:648] HybridClock initialized: now 1781997104884506 us; error 28 us; skew 500 ppm
I20260620 23:11:44.884886 18574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:11:44 dist-test-slave-jkbq krb5kdc[18561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997104, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.191.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:44.892976 18574 init.cc:377] Logged in from keytab as oryx/127.10.191.254@KRBTEST.COM (short username oryx)
I20260620 23:11:44.894536 18574 webserver.cc:460] Webserver started at http://127.10.191.254:33933/ using document root <none> and password file <none>
I20260620 23:11:44.895293 18574 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:44.895385 18574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:44.895651 18574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:11:44.898470 18574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "61bc181e1aab415b945d91121993c454"
format_stamp: "Formatted at 2026-06-20 23:11:44 on dist-test-slave-jkbq"
server_key: "606f0698ceda640458ba6ecb4b1f5995"
server_key_iv: "6e64bb8df222f8b1b6e2fe416a40d4d0"
server_key_version: "encryptionkey@0"
I20260620 23:11:44.899273 18574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "61bc181e1aab415b945d91121993c454"
format_stamp: "Formatted at 2026-06-20 23:11:44 on dist-test-slave-jkbq"
server_key: "606f0698ceda640458ba6ecb4b1f5995"
server_key_iv: "6e64bb8df222f8b1b6e2fe416a40d4d0"
server_key_version: "encryptionkey@0"
I20260620 23:11:44.904139 18574 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 23:11:44.907518 18589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:44.908815 18574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:11:44.908993 18574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "61bc181e1aab415b945d91121993c454"
format_stamp: "Formatted at 2026-06-20 23:11:44 on dist-test-slave-jkbq"
server_key: "606f0698ceda640458ba6ecb4b1f5995"
server_key_iv: "6e64bb8df222f8b1b6e2fe416a40d4d0"
server_key_version: "encryptionkey@0"
I20260620 23:11:44.909194 18574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:44.928998 18574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:44.933251 18574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:44.933522 18574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:44.954842 18574 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:36897
I20260620 23:11:44.954819 18650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:36897 every 8 connection(s)
I20260620 23:11:44.956194 18574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:11:44.959829 18651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:11:44.962641 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 18574
I20260620 23:11:44.962841 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:11:44.963661 11007 external_mini_cluster.cc:1428] Setting key 4a452cb2e4f04e2e729044e1613573bf
I20260620 23:11:44.972278 18651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61bc181e1aab415b945d91121993c454: Bootstrap starting.
I20260620 23:11:44.976003 18651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61bc181e1aab415b945d91121993c454: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:44.977290 18651 log.cc:824] T 00000000000000000000000000000000 P 61bc181e1aab415b945d91121993c454: Log is configured to *not* fsync() on all Append() calls
Jun 20 23:11:44 dist-test-slave-jkbq krb5kdc[18561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM
I20260620 23:11:44.980103 18651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61bc181e1aab415b945d91121993c454: No bootstrap required, opened a new log
I20260620 23:11:44.986533 18654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:44.965732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57978 (local address 127.10.191.254:36897)
0620 23:11:44.966548 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:44.966568 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:44.966609 (+ 41us) server_negotiation.cc:407] Connection header received
0620 23:11:44.968341 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:44.968364 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:44.968761 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:44.969080 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:44.970782 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:44.972111 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:44.973860 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:44.974512 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:44.974649 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:44.979424 (+ 4775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:44.979431 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:44.979445 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:44.979481 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:44.982600 (+ 3119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:44.983346 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:44.983352 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:44.983358 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:44.983518 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:44.983892 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:44.983900 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:44.983910 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:44.984415 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:44.984778 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:44.985164 (+ 386us) server_negotiation.cc:299] Negotiation successful
0620 23:11:44.985576 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":387,"threads_started":1}
I20260620 23:11:44.988128 18651 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61bc181e1aab415b945d91121993c454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61bc181e1aab415b945d91121993c454" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 36897 } }
I20260620 23:11:44.988471 18651 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61bc181e1aab415b945d91121993c454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:44.988582 18651 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61bc181e1aab415b945d91121993c454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61bc181e1aab415b945d91121993c454, State: Initialized, Role: FOLLOWER
I20260620 23:11:44.989217 18651 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61bc181e1aab415b945d91121993c454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61bc181e1aab415b945d91121993c454" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 36897 } }
I20260620 23:11:44.989470 18651 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61bc181e1aab415b945d91121993c454 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:11:44.989594 18651 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61bc181e1aab415b945d91121993c454 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:11:44.989773 18651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61bc181e1aab415b945d91121993c454 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:44.991863 18651 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61bc181e1aab415b945d91121993c454 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61bc181e1aab415b945d91121993c454" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 36897 } }
I20260620 23:11:44.992327 18651 leader_election.cc:302] T 00000000000000000000000000000000 P 61bc181e1aab415b945d91121993c454 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61bc181e1aab415b945d91121993c454; no voters:
I20260620 23:11:44.992828 18651 leader_election.cc:288] T 00000000000000000000000000000000 P 61bc181e1aab415b945d91121993c454 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:11:44.992954 18656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61bc181e1aab415b945d91121993c454 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:11:44.993631 18656 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61bc181e1aab415b945d91121993c454 [term 1 LEADER]: Becoming Leader. State: Replica: 61bc181e1aab415b945d91121993c454, State: Running, Role: LEADER
I20260620 23:11:44.994139 18656 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61bc181e1aab415b945d91121993c454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61bc181e1aab415b945d91121993c454" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 36897 } }
I20260620 23:11:44.994771 18651 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61bc181e1aab415b945d91121993c454 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:11:44.997092 18657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61bc181e1aab415b945d91121993c454 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61bc181e1aab415b945d91121993c454" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61bc181e1aab415b945d91121993c454" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 36897 } } }
I20260620 23:11:44.996909 18658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61bc181e1aab415b945d91121993c454 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61bc181e1aab415b945d91121993c454. Latest consensus state: current_term: 1 leader_uuid: "61bc181e1aab415b945d91121993c454" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61bc181e1aab415b945d91121993c454" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 36897 } } }
I20260620 23:11:44.997407 18657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61bc181e1aab415b945d91121993c454 [sys.catalog]: This master's current role is: LEADER
I20260620 23:11:44.997407 18658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61bc181e1aab415b945d91121993c454 [sys.catalog]: This master's current role is: LEADER
I20260620 23:11:44.998435 18664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:11:45.004038 18664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:11:45.012207 18664 catalog_manager.cc:1347] Generated new cluster ID: 4734a4c1404e4cecb1d8e3721a7a26ca
I20260620 23:11:45.012341 18664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:11:45.051721 18664 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:11:45.053082 18664 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:11:45.062206 18664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61bc181e1aab415b945d91121993c454: Generated new TSK 0
I20260620 23:11:45.063133 18664 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.10.191.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.191.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.191.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.193.keytab.
Entry for principal oryx/127.10.191.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.193.keytab.
I20260620 23:11:45.118178 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.193:0
--local_ip_for_outbound_sockets=127.10.191.193
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:36897
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.193.keytab
--principal=oryx/127.10.191.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:42351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:11:45.456679 18677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:45.456960 18677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:45.457056 18677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:45.464210 18677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:45.464315 18677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:45.464457 18677 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:11:45.464592 18677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.193
I20260620 23:11:45.474249 18677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:42351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.193.keytab
--principal=oryx/127.10.191.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.191.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.191.254:36897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.193
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:45.475692 18677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:45.477020 18677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:45.490315 18682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:45.490315 18685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:45.490362 18683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:45.491255 18677 server_base.cc:1034] running on GCE node
I20260620 23:11:45.491803 18677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:45.492635 18677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:45.493877 18677 hybrid_clock.cc:648] HybridClock initialized: now 1781997105493791 us; error 56 us; skew 500 ppm
I20260620 23:11:45.494244 18677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:11:45 dist-test-slave-jkbq krb5kdc[18561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997105, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.191.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:45.497717 18677 init.cc:377] Logged in from keytab as oryx/127.10.191.193@KRBTEST.COM (short username oryx)
I20260620 23:11:45.499297 18677 webserver.cc:460] Webserver started at http://127.10.191.193:32877/ using document root <none> and password file <none>
I20260620 23:11:45.500028 18677 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:45.500121 18677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:45.500385 18677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:11:45.503345 18677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-0/data/instance:
uuid: "e94d40b28f5d4842819973b4fb7b8272"
format_stamp: "Formatted at 2026-06-20 23:11:45 on dist-test-slave-jkbq"
server_key: "b7b64f4fdf080fce54486032aa8f0f8d"
server_key_iv: "97a067f2583a6ef244e85e74067fd870"
server_key_version: "encryptionkey@0"
I20260620 23:11:45.504272 18677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance:
uuid: "e94d40b28f5d4842819973b4fb7b8272"
format_stamp: "Formatted at 2026-06-20 23:11:45 on dist-test-slave-jkbq"
server_key: "b7b64f4fdf080fce54486032aa8f0f8d"
server_key_iv: "97a067f2583a6ef244e85e74067fd870"
server_key_version: "encryptionkey@0"
I20260620 23:11:45.509851 18677 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:11:45.513409 18692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:45.514791 18677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 23:11:45.515002 18677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-0/wal
uuid: "e94d40b28f5d4842819973b4fb7b8272"
format_stamp: "Formatted at 2026-06-20 23:11:45 on dist-test-slave-jkbq"
server_key: "b7b64f4fdf080fce54486032aa8f0f8d"
server_key_iv: "97a067f2583a6ef244e85e74067fd870"
server_key_version: "encryptionkey@0"
I20260620 23:11:45.515226 18677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:45.531458 18677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:45.535362 18677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:45.535636 18677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:45.538134 18677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:11:45.538234 18677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:45.538360 18677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:11:45.538432 18677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:11:45 dist-test-slave-jkbq krb5kdc[18561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997105, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.191.193@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM
I20260620 23:11:45.562343 18701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:45.538801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:52911 (local address 127.10.191.254:36897)
0620 23:11:45.539411 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:45.539415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:45.540518 (+ 1103us) server_negotiation.cc:407] Connection header received
0620 23:11:45.542504 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:45.542509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:45.542610 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:45.542731 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:45.544882 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:45.545878 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:45.549262 (+ 3384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:45.549791 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:45.549940 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:45.554979 (+ 5039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:45.554987 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:45.554994 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:45.555044 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:45.558262 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:45.559263 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:45.559270 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:45.559274 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:45.559393 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:45.559996 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:45.560005 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:45.560011 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:45.560482 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:45.560681 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:45.561427 (+ 746us) server_negotiation.cc:299] Negotiation successful
0620 23:11:45.561721 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":419,"threads_started":1}
I20260620 23:11:45.563146 18702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:45.539504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:36897 (local address 127.10.191.193:52911)
0620 23:11:45.540367 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:45.540404 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:45.542227 (+ 1823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:45.543002 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:45.543035 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:45.543806 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:45.544710 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:45.544729 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:45.546091 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:45.546100 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:45.549068 (+ 2968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:45.549108 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:45.550048 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:45.550059 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:45.550429 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:45.551379 (+ 950us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:45.551417 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:45.554643 (+ 3226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:45.558386 (+ 3743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:45.558394 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:45.558414 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:45.559126 (+ 712us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:45.559501 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:45.559507 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:45.559514 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:45.559824 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:45.560642 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:45.560648 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:45.560973 (+ 325us) client_negotiation.cc:769] Sending connection context
0620 23:11:45.561285 (+ 312us) client_negotiation.cc:240] Negotiation successful
0620 23:11:45.561723 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":353,"threads_started":1}
I20260620 23:11:45.583333 18701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:45.568808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:60313 (local address 127.10.191.254:36897)
0620 23:11:45.569019 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:45.569028 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:45.569770 (+ 742us) server_negotiation.cc:407] Connection header received
0620 23:11:45.569905 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:45.569912 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:45.570012 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:45.570141 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:45.571068 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:45.571804 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:45.573943 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:45.574547 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:45.574619 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:45.577262 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:45.577270 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:45.577277 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:45.577316 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:45.580427 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:45.581379 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:45.581387 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:45.581391 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:45.581553 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:45.581907 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:45.581912 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:45.581916 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:45.582173 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:45.582323 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:45.582738 (+ 415us) server_negotiation.cc:299] Negotiation successful
0620 23:11:45.582986 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 23:11:45.583616 18766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:45.568970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:36897 (local address 127.10.191.193:60313)
0620 23:11:45.569663 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:45.569682 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:45.569793 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:45.570159 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:45.570161 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:45.570577 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:45.570920 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:45.570934 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:45.571965 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:45.571970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:45.573726 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:45.573755 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:45.574934 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:45.574943 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:45.575154 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:45.575825 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:45.575853 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:45.577028 (+ 1175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:45.580535 (+ 3507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:45.580542 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:45.580546 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:45.581256 (+ 710us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:45.581660 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:45.581665 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:45.581669 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:45.581801 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:45.582295 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:45.582300 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:45.582481 (+ 181us) client_negotiation.cc:769] Sending connection context
0620 23:11:45.582631 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 23:11:45.582996 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":388,"threads_started":1}
I20260620 23:11:45.591053 18677 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.193:37597
I20260620 23:11:45.591074 18812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.193:37597 every 8 connection(s)
I20260620 23:11:45.592213 18677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
I20260620 23:11:45.594161 18813 heartbeater.cc:347] Connected to a master server at 127.10.191.254:36897
I20260620 23:11:45.594372 18813 heartbeater.cc:464] Registering TS with master...
I20260620 23:11:45.595131 18813 heartbeater.cc:511] Master 127.10.191.254:36897 requested a full tablet report, sending...
I20260620 23:11:45.597028 18606 ts_manager.cc:194] Registered new tserver with Master: e94d40b28f5d4842819973b4fb7b8272 (127.10.191.193:37597)
I20260620 23:11:45.598423 18606 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.191.193@KRBTEST.COM'} at 127.10.191.193:52911
I20260620 23:11:45.601397 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 18677
I20260620 23:11:45.601585 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance
I20260620 23:11:45.602221 11007 external_mini_cluster.cc:1428] Setting key 9d9c6565f52225e47e624a1880a525a7
WARNING: no policy specified for oryx/127.10.191.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.191.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.191.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.194.keytab.
Entry for principal oryx/127.10.191.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.194.keytab.
I20260620 23:11:45.646204 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.194:0
--local_ip_for_outbound_sockets=127.10.191.194
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:36897
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.194.keytab
--principal=oryx/127.10.191.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:42351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:11:45.963551 18817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:45.963841 18817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:45.964007 18817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:45.971392 18817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:45.971531 18817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:45.971668 18817 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:11:45.971798 18817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.194
I20260620 23:11:45.981220 18817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:42351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.194.keytab
--principal=oryx/127.10.191.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.191.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.191.254:36897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.194
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:45.982537 18817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:45.983865 18817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:45.997032 18822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:45.997457 18823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:45.997534 18825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:45.997637 18817 server_base.cc:1034] running on GCE node
I20260620 23:11:45.998354 18817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:45.999396 18817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:46.000631 18817 hybrid_clock.cc:648] HybridClock initialized: now 1781997106000552 us; error 66 us; skew 500 ppm
I20260620 23:11:46.000941 18817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:11:45 dist-test-slave-jkbq krb5kdc[18561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997105, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.191.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:46.003963 18817 init.cc:377] Logged in from keytab as oryx/127.10.191.194@KRBTEST.COM (short username oryx)
I20260620 23:11:46.005287 18817 webserver.cc:460] Webserver started at http://127.10.191.194:46325/ using document root <none> and password file <none>
I20260620 23:11:46.005975 18817 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:46.006104 18817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:46.006420 18817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:11:46.009240 18817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-1/data/instance:
uuid: "bfe128cc990a400b88234db04775deec"
format_stamp: "Formatted at 2026-06-20 23:11:45 on dist-test-slave-jkbq"
server_key: "a0d29e0c55e5eb767b1b1d0410930991"
server_key_iv: "8bcf4f42c92b4a79b4278f90469642e0"
server_key_version: "encryptionkey@0"
I20260620 23:11:46.010005 18817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance:
uuid: "bfe128cc990a400b88234db04775deec"
format_stamp: "Formatted at 2026-06-20 23:11:45 on dist-test-slave-jkbq"
server_key: "a0d29e0c55e5eb767b1b1d0410930991"
server_key_iv: "8bcf4f42c92b4a79b4278f90469642e0"
server_key_version: "encryptionkey@0"
I20260620 23:11:46.015072 18817 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 23:11:46.018476 18832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:46.019878 18817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:11:46.020048 18817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-1/wal
uuid: "bfe128cc990a400b88234db04775deec"
format_stamp: "Formatted at 2026-06-20 23:11:45 on dist-test-slave-jkbq"
server_key: "a0d29e0c55e5eb767b1b1d0410930991"
server_key_iv: "8bcf4f42c92b4a79b4278f90469642e0"
server_key_version: "encryptionkey@0"
I20260620 23:11:46.020258 18817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:46.040876 18817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:46.044725 18817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:46.044989 18817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:46.047484 18817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:11:46.047605 18817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:46.047755 18817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:11:46.047825 18817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:11:46 dist-test-slave-jkbq krb5kdc[18561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997105, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.191.194@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM
I20260620 23:11:46.071914 18701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:46.048297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:48529 (local address 127.10.191.254:36897)
0620 23:11:46.048488 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:46.048492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:46.049948 (+ 1456us) server_negotiation.cc:407] Connection header received
0620 23:11:46.052006 (+ 2058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:46.052012 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:46.052135 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:46.052251 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:46.054553 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:46.055606 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:46.058890 (+ 3284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:46.059436 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:46.059578 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:46.065283 (+ 5705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:46.065292 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:46.065299 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:46.065350 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:46.068795 (+ 3445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:46.069506 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:46.069512 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:46.069516 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:46.069639 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:46.070073 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:46.070080 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:46.070083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:46.070357 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:46.070516 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:46.071340 (+ 824us) server_negotiation.cc:299] Negotiation successful
0620 23:11:46.071561 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 23:11:46.073407 18841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:46.048937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:36897 (local address 127.10.191.194:48529)
0620 23:11:46.049761 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:46.049825 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:46.051734 (+ 1909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:46.052571 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:46.052586 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:46.053419 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:46.054291 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:46.054310 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:46.055777 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:46.055785 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:46.058621 (+ 2836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:46.058640 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:46.059759 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:46.059770 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:46.060179 (+ 409us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:46.061079 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:46.061125 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:46.065032 (+ 3907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:46.068934 (+ 3902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:46.068943 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:46.068959 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:46.069378 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:46.069759 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:46.069764 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:46.069767 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:46.069963 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:46.070494 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:46.070503 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:46.070857 (+ 354us) client_negotiation.cc:769] Sending connection context
0620 23:11:46.071240 (+ 383us) client_negotiation.cc:240] Negotiation successful
0620 23:11:46.071764 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":355,"threads_started":1}
I20260620 23:11:46.098275 18701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:46.081837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:50507 (local address 127.10.191.254:36897)
0620 23:11:46.082060 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:46.082068 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:46.083263 (+ 1195us) server_negotiation.cc:407] Connection header received
0620 23:11:46.083336 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:46.083344 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:46.083428 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:46.083532 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:46.084520 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:46.086038 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:46.087605 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:46.087991 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:46.088111 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:46.090793 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:46.090799 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:46.090803 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:46.090837 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:46.094611 (+ 3774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:46.095789 (+ 1178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:46.095794 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:46.095798 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:46.095987 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:46.096529 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:46.096537 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:46.096541 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:46.097012 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:46.097191 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:46.097650 (+ 459us) server_negotiation.cc:299] Negotiation successful
0620 23:11:46.097872 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 23:11:46.098299 18914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:46.081963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:36897 (local address 127.10.191.194:50507)
0620 23:11:46.082912 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:46.082935 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:46.083112 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:46.083587 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:46.083592 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:46.083998 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:46.084374 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:46.084384 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:46.086182 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:46.086186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:46.087406 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:46.087423 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:46.088430 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:46.088435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:46.088570 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:46.089387 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:46.089403 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:46.090549 (+ 1146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:46.094784 (+ 4235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:46.094793 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:46.094799 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:46.095570 (+ 771us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:46.096136 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:46.096142 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:46.096148 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:46.096353 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:46.097188 (+ 835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:46.097195 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:46.097389 (+ 194us) client_negotiation.cc:769] Sending connection context
0620 23:11:46.097497 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 23:11:46.097734 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"thread_start_us":368,"threads_started":1}
I20260620 23:11:46.103729 18817 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.194:37179
I20260620 23:11:46.103758 18951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.194:37179 every 8 connection(s)
I20260620 23:11:46.104928 18817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
I20260620 23:11:46.105423 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 18817
I20260620 23:11:46.105614 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance
I20260620 23:11:46.106196 11007 external_mini_cluster.cc:1428] Setting key 8af8b4267fcfc15c5131372e3ab923bb
I20260620 23:11:46.107041 18952 heartbeater.cc:347] Connected to a master server at 127.10.191.254:36897
I20260620 23:11:46.107293 18952 heartbeater.cc:464] Registering TS with master...
I20260620 23:11:46.107990 18952 heartbeater.cc:511] Master 127.10.191.254:36897 requested a full tablet report, sending...
I20260620 23:11:46.109244 18606 ts_manager.cc:194] Registered new tserver with Master: bfe128cc990a400b88234db04775deec (127.10.191.194:37179)
I20260620 23:11:46.110023 18606 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.191.194@KRBTEST.COM'} at 127.10.191.194:48529
WARNING: no policy specified for oryx/127.10.191.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.191.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.191.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.195.keytab.
Entry for principal oryx/127.10.191.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.195.keytab.
I20260620 23:11:46.150327 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.195:0
--local_ip_for_outbound_sockets=127.10.191.195
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:36897
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.195.keytab
--principal=oryx/127.10.191.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:42351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:11:46.473132 18956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:46.473412 18956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:46.473508 18956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:46.480830 18956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:46.480974 18956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:46.481128 18956 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:11:46.481277 18956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.195
I20260620 23:11:46.490756 18956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:42351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.195.keytab
--principal=oryx/127.10.191.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.191.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.191.254:36897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.195
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:46.492025 18956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:46.493340 18956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:46.506703 18961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:46.506843 18962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:46.506889 18964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:46.508389 18956 server_base.cc:1034] running on GCE node
I20260620 23:11:46.509109 18956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:46.510159 18956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:46.511363 18956 hybrid_clock.cc:648] HybridClock initialized: now 1781997106511289 us; error 59 us; skew 500 ppm
I20260620 23:11:46.511620 18956 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:11:46 dist-test-slave-jkbq krb5kdc[18561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997106, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.191.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:46.514915 18956 init.cc:377] Logged in from keytab as oryx/127.10.191.195@KRBTEST.COM (short username oryx)
I20260620 23:11:46.516309 18956 webserver.cc:460] Webserver started at http://127.10.191.195:39111/ using document root <none> and password file <none>
I20260620 23:11:46.517011 18956 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:46.517122 18956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:46.517450 18956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:11:46.520426 18956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-2/data/instance:
uuid: "663f3caaf3a24449ae9075cd10afb8c7"
format_stamp: "Formatted at 2026-06-20 23:11:46 on dist-test-slave-jkbq"
server_key: "3326cea97eb5d5a64b10671991c00a5e"
server_key_iv: "8278aaaf30e7557f2ac4888c55c01dcf"
server_key_version: "encryptionkey@0"
I20260620 23:11:46.521196 18956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance:
uuid: "663f3caaf3a24449ae9075cd10afb8c7"
format_stamp: "Formatted at 2026-06-20 23:11:46 on dist-test-slave-jkbq"
server_key: "3326cea97eb5d5a64b10671991c00a5e"
server_key_iv: "8278aaaf30e7557f2ac4888c55c01dcf"
server_key_version: "encryptionkey@0"
I20260620 23:11:46.526201 18956 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 23:11:46.529598 18971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:46.531085 18956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:11:46.531253 18956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-2/wal
uuid: "663f3caaf3a24449ae9075cd10afb8c7"
format_stamp: "Formatted at 2026-06-20 23:11:46 on dist-test-slave-jkbq"
server_key: "3326cea97eb5d5a64b10671991c00a5e"
server_key_iv: "8278aaaf30e7557f2ac4888c55c01dcf"
server_key_version: "encryptionkey@0"
I20260620 23:11:46.531489 18956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:46.552222 18956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:46.556272 18956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:46.556543 18956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:46.559389 18956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:11:46.559471 18956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:46.559619 18956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:11:46.559701 18956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:11:46 dist-test-slave-jkbq krb5kdc[18561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997106, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.191.195@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM
I20260620 23:11:46.584656 18701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:46.560277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:46445 (local address 127.10.191.254:36897)
0620 23:11:46.560472 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:46.560477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:46.562046 (+ 1569us) server_negotiation.cc:407] Connection header received
0620 23:11:46.564396 (+ 2350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:46.564402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:46.564505 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:46.564625 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:46.567529 (+ 2904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:46.568325 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:46.571529 (+ 3204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:46.572006 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:46.572171 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:46.576990 (+ 4819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:46.576998 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:46.577005 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:46.577052 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:46.580839 (+ 3787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:46.581832 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:46.581836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:46.581839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:46.582018 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:46.582677 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:46.582684 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:46.582690 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:46.583046 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:46.583220 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:46.584059 (+ 839us) server_negotiation.cc:299] Negotiation successful
0620 23:11:46.584298 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 23:11:46.585659 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:46.560951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:36897 (local address 127.10.191.195:46445)
0620 23:11:46.561862 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:46.561901 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:46.564069 (+ 2168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:46.565058 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:46.565071 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:46.566143 (+ 1072us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:46.567272 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:46.567305 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:46.568493 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:46.568501 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:46.571343 (+ 2842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:46.571362 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:46.572299 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:46.572310 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:46.572656 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:46.573536 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:46.573564 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:46.576694 (+ 3130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:46.581008 (+ 4314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:46.581021 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:46.581037 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:46.581641 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:46.582186 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:46.582193 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:46.582199 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:46.582478 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:46.583197 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:46.583205 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:46.583586 (+ 381us) client_negotiation.cc:769] Sending connection context
0620 23:11:46.583940 (+ 354us) client_negotiation.cc:240] Negotiation successful
0620 23:11:46.584454 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":396,"threads_started":1}
I20260620 23:11:46.601033 18813 heartbeater.cc:503] Master 127.10.191.254:36897 was elected leader, sending a full tablet report...
I20260620 23:11:46.605010 19038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:46.591526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:36897 (local address 127.10.191.195:59841)
0620 23:11:46.592147 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:46.592163 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:46.592310 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:46.592617 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:46.592622 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:46.593055 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:46.593329 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:46.593338 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:46.594342 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:46.594346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:46.595514 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:46.595525 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:46.596421 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:46.596427 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:46.596598 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:46.597270 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:46.597302 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:46.598027 (+ 725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:46.601784 (+ 3757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:46.601788 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:46.601791 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:46.602316 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:46.602775 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:46.602779 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:46.602782 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:46.602958 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:46.603563 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:46.603569 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:46.603786 (+ 217us) client_negotiation.cc:769] Sending connection context
0620 23:11:46.603940 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 23:11:46.604306 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":331,"threads_started":1}
I20260620 23:11:46.605774 18701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:46.591512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:59841 (local address 127.10.191.254:36897)
0620 23:11:46.591706 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:46.591711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:46.592246 (+ 535us) server_negotiation.cc:407] Connection header received
0620 23:11:46.592421 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:46.592427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:46.592490 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:46.592591 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:46.593466 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:46.594137 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:46.595650 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:46.596220 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:46.596378 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:46.598264 (+ 1886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:46.598270 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:46.598275 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:46.598313 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:46.601645 (+ 3332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:46.602470 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:46.602475 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:46.602478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:46.602638 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:46.603098 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:46.603104 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:46.603120 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:46.603436 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:46.604929 (+ 1493us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:46.605071 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 23:11:46.605444 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 23:11:46.619558 18956 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.195:42731
I20260620 23:11:46.619638 19090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.195:42731 every 8 connection(s)
I20260620 23:11:46.620752 18956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
I20260620 23:11:46.622846 19091 heartbeater.cc:347] Connected to a master server at 127.10.191.254:36897
I20260620 23:11:46.623147 19091 heartbeater.cc:464] Registering TS with master...
I20260620 23:11:46.623853 19091 heartbeater.cc:511] Master 127.10.191.254:36897 requested a full tablet report, sending...
I20260620 23:11:46.625186 18606 ts_manager.cc:194] Registered new tserver with Master: 663f3caaf3a24449ae9075cd10afb8c7 (127.10.191.195:42731)
I20260620 23:11:46.625993 18606 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.191.195@KRBTEST.COM'} at 127.10.191.195:46445
I20260620 23:11:46.630991 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 18956
I20260620 23:11:46.631153 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance
I20260620 23:11:46.631582 11007 external_mini_cluster.cc:1428] Setting key 190ce483549fff8c613a4d33bbea2074
I20260620 23:11:46.634339 11007 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 23:11:46 dist-test-slave-jkbq krb5kdc[18561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997106, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 23:11:46 dist-test-slave-jkbq krb5kdc[18561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:11:46 dist-test-slave-jkbq krb5kdc[18561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 23:11:46.681090 18701 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:11:46.670812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57980 (local address 127.10.191.254:36897)
0620 23:11:46.671023 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:46.671027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:46.671392 (+ 365us) server_negotiation.cc:407] Connection header received
0620 23:11:46.671542 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:46.671546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:46.671612 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:46.671707 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:46.672940 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:46.673731 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:46.675119 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:46.675543 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:46.675712 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:46.680882 (+ 5170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57980: BlockingRecv error: recv got EOF from 127.0.0.1:57980 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Jun 20 23:11:46 dist-test-slave-jkbq krb5kdc[18561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997106, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM
I20260620 23:11:46.697036 18701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:46.684282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57988 (local address 127.10.191.254:36897)
0620 23:11:46.684494 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:46.684502 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:46.684794 (+ 292us) server_negotiation.cc:407] Connection header received
0620 23:11:46.684990 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:46.684995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:46.685078 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:46.685191 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:46.686305 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:46.687067 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:46.688266 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:46.688698 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:46.688820 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:46.692229 (+ 3409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:46.692234 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:46.692238 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:46.692266 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:46.694513 (+ 2247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:46.695225 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:46.695232 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:46.695236 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:46.695364 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:46.695712 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:46.695719 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:46.695722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:46.695968 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:46.696232 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:46.696485 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 23:11:46.696699 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 23:11:46.701097 18606 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:11:46.707688 18606 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:11:46.740394 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:46.719263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.195:42731 (local address 127.0.0.1:51996)
0620 23:11:46.730433 (+ 11170us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:46.730481 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:46.730657 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:46.731966 (+ 1309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:46.731975 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:46.732007 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:11:46.732448 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:46.732461 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:46.734831 (+ 2370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:46.734836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:46.737510 (+ 2674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:46.737527 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:46.739395 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:46.739403 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:46.739586 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:46.739652 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 23:11:46.739753 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 23:11:46.739947 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10941,"mutex_wait_us":132,"thread_start_us":12150,"threads_started":1}
I20260620 23:11:46.740957 19112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:46.720059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55916 (local address 127.10.191.194:37179)
0620 23:11:46.731259 (+ 11200us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:46.731267 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:46.732318 (+ 1051us) server_negotiation.cc:407] Connection header received
0620 23:11:46.732449 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:46.732460 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:46.732742 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:46.732953 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:11:46.733584 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:46.735082 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:46.738106 (+ 3024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:46.739446 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:46.739595 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:46.739769 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:46.740488 (+ 719us) server_negotiation.cc:299] Negotiation successful
0620 23:11:46.740650 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11037,"thread_start_us":10812,"threads_started":1}
I20260620 23:11:46.741544 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:46.719617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51996 (local address 127.10.191.195:42731)
0620 23:11:46.731336 (+ 11719us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:46.731342 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:46.731368 (+ 26us) server_negotiation.cc:407] Connection header received
0620 23:11:46.731441 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:46.731447 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:46.731731 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:46.732669 (+ 938us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:11:46.732943 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:46.734687 (+ 1744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:46.737700 (+ 3013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:46.739167 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:46.740949 (+ 1782us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:46.741133 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:46.741200 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 23:11:46.741324 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11557,"thread_start_us":11389,"threads_started":1}
I20260620 23:11:46.742048 19109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:46.719431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.194:37179 (local address 127.0.0.1:55916)
0620 23:11:46.731709 (+ 12278us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:46.731732 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:46.731847 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:46.732954 (+ 1107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:46.732958 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:46.732986 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:11:46.733383 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:46.733398 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:46.735252 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:46.735263 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:46.737937 (+ 2674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:46.737949 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:46.739915 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:46.739921 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:46.740097 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:46.740140 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 23:11:46.740227 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 23:11:46.740352 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12119,"mutex_wait_us":3,"thread_start_us":10618,"threads_started":1}
I20260620 23:11:46.746026 18882 tablet_service.cc:1467] Processing CreateTablet for tablet 2cb0b019108e49ed88a08ee7bc5bbb68 (DEFAULT_TABLE table=test-table [id=5a7802845c174f2694f2397cf6ef489b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:11:46.746064 19021 tablet_service.cc:1467] Processing CreateTablet for tablet 2cb0b019108e49ed88a08ee7bc5bbb68 (DEFAULT_TABLE table=test-table [id=5a7802845c174f2694f2397cf6ef489b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:11:46.747394 19021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cb0b019108e49ed88a08ee7bc5bbb68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:11:46.747414 18882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cb0b019108e49ed88a08ee7bc5bbb68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:11:46.749943 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:46.733330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:37597 (local address 127.0.0.1:41416)
0620 23:11:46.740613 (+ 7283us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:46.740641 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:46.740728 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:46.743502 (+ 2774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:46.743507 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:46.743533 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:11:46.743691 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:46.743701 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:46.745603 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:46.745608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:46.747398 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:46.747413 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:46.749188 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:46.749194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:46.749376 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:46.749408 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 23:11:46.749503 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 23:11:46.749638 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7192,"thread_start_us":9921,"threads_started":1}
I20260620 23:11:46.750042 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:46.732033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41416 (local address 127.10.191.193:37597)
0620 23:11:46.742605 (+ 10572us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:46.742612 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:46.742646 (+ 34us) server_negotiation.cc:407] Connection header received
0620 23:11:46.742735 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:46.742742 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:46.742988 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:46.743856 (+ 868us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:11:46.744061 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:46.745446 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:46.747581 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:46.749001 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:46.749181 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:46.749323 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:46.749563 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 23:11:46.749743 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10391,"thread_start_us":10260,"threads_started":1}
I20260620 23:11:46.753571 18743 tablet_service.cc:1467] Processing CreateTablet for tablet 2cb0b019108e49ed88a08ee7bc5bbb68 (DEFAULT_TABLE table=test-table [id=5a7802845c174f2694f2397cf6ef489b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:11:46.754925 18743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cb0b019108e49ed88a08ee7bc5bbb68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:11:46.758198 19115 tablet_bootstrap.cc:492] T 2cb0b019108e49ed88a08ee7bc5bbb68 P bfe128cc990a400b88234db04775deec: Bootstrap starting.
I20260620 23:11:46.761467 19115 tablet_bootstrap.cc:654] T 2cb0b019108e49ed88a08ee7bc5bbb68 P bfe128cc990a400b88234db04775deec: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:46.761817 19116 tablet_bootstrap.cc:492] T 2cb0b019108e49ed88a08ee7bc5bbb68 P 663f3caaf3a24449ae9075cd10afb8c7: Bootstrap starting.
I20260620 23:11:46.762923 19115 log.cc:824] T 2cb0b019108e49ed88a08ee7bc5bbb68 P bfe128cc990a400b88234db04775deec: Log is configured to *not* fsync() on all Append() calls
I20260620 23:11:46.765061 19116 tablet_bootstrap.cc:654] T 2cb0b019108e49ed88a08ee7bc5bbb68 P 663f3caaf3a24449ae9075cd10afb8c7: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:46.765961 19115 tablet_bootstrap.cc:492] T 2cb0b019108e49ed88a08ee7bc5bbb68 P bfe128cc990a400b88234db04775deec: No bootstrap required, opened a new log
I20260620 23:11:46.766220 19115 ts_tablet_manager.cc:1397] T 2cb0b019108e49ed88a08ee7bc5bbb68 P bfe128cc990a400b88234db04775deec: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.004s
I20260620 23:11:46.766275 19116 log.cc:824] T 2cb0b019108e49ed88a08ee7bc5bbb68 P 663f3caaf3a24449ae9075cd10afb8c7: Log is configured to *not* fsync() on all Append() calls
I20260620 23:11:46.768494 19119 tablet_bootstrap.cc:492] T 2cb0b019108e49ed88a08ee7bc5bbb68 P e94d40b28f5d4842819973b4fb7b8272: Bootstrap starting.
I20260620 23:11:46.769098 19116 tablet_bootstrap.cc:492] T 2cb0b019108e49ed88a08ee7bc5bbb68 P 663f3caaf3a24449ae9075cd10afb8c7: No bootstrap required, opened a new log
I20260620 23:11:46.769348 19116 ts_tablet_manager.cc:1397] T 2cb0b019108e49ed88a08ee7bc5bbb68 P 663f3caaf3a24449ae9075cd10afb8c7: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260620 23:11:46.771663 19119 tablet_bootstrap.cc:654] T 2cb0b019108e49ed88a08ee7bc5bbb68 P e94d40b28f5d4842819973b4fb7b8272: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:46.772590 19119 log.cc:824] T 2cb0b019108e49ed88a08ee7bc5bbb68 P e94d40b28f5d4842819973b4fb7b8272: Log is configured to *not* fsync() on all Append() calls
I20260620 23:11:46.775074 19119 tablet_bootstrap.cc:492] T 2cb0b019108e49ed88a08ee7bc5bbb68 P e94d40b28f5d4842819973b4fb7b8272: No bootstrap required, opened a new log
I20260620 23:11:46.775327 19119 ts_tablet_manager.cc:1397] T 2cb0b019108e49ed88a08ee7bc5bbb68 P e94d40b28f5d4842819973b4fb7b8272: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260620 23:11:46.776579 19115 raft_consensus.cc:348] T 2cb0b019108e49ed88a08ee7bc5bbb68 P bfe128cc990a400b88234db04775deec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } }
I20260620 23:11:46.777005 19115 raft_consensus.cc:374] T 2cb0b019108e49ed88a08ee7bc5bbb68 P bfe128cc990a400b88234db04775deec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:46.777092 19115 raft_consensus.cc:729] T 2cb0b019108e49ed88a08ee7bc5bbb68 P bfe128cc990a400b88234db04775deec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfe128cc990a400b88234db04775deec, State: Initialized, Role: FOLLOWER
I20260620 23:11:46.777683 19115 consensus_queue.cc:260] T 2cb0b019108e49ed88a08ee7bc5bbb68 P bfe128cc990a400b88234db04775deec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } }
I20260620 23:11:46.778751 18952 heartbeater.cc:503] Master 127.10.191.254:36897 was elected leader, sending a full tablet report...
I20260620 23:11:46.779652 19116 raft_consensus.cc:348] T 2cb0b019108e49ed88a08ee7bc5bbb68 P 663f3caaf3a24449ae9075cd10afb8c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } }
I20260620 23:11:46.780047 19116 raft_consensus.cc:374] T 2cb0b019108e49ed88a08ee7bc5bbb68 P 663f3caaf3a24449ae9075cd10afb8c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:46.780146 19116 raft_consensus.cc:729] T 2cb0b019108e49ed88a08ee7bc5bbb68 P 663f3caaf3a24449ae9075cd10afb8c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663f3caaf3a24449ae9075cd10afb8c7, State: Initialized, Role: FOLLOWER
I20260620 23:11:46.780754 19115 ts_tablet_manager.cc:1428] T 2cb0b019108e49ed88a08ee7bc5bbb68 P bfe128cc990a400b88234db04775deec: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260620 23:11:46.780759 19116 consensus_queue.cc:260] T 2cb0b019108e49ed88a08ee7bc5bbb68 P 663f3caaf3a24449ae9075cd10afb8c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } }
I20260620 23:11:46.781836 19091 heartbeater.cc:503] Master 127.10.191.254:36897 was elected leader, sending a full tablet report...
I20260620 23:11:46.782450 19116 ts_tablet_manager.cc:1428] T 2cb0b019108e49ed88a08ee7bc5bbb68 P 663f3caaf3a24449ae9075cd10afb8c7: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260620 23:11:46.783842 19119 raft_consensus.cc:348] T 2cb0b019108e49ed88a08ee7bc5bbb68 P e94d40b28f5d4842819973b4fb7b8272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } }
I20260620 23:11:46.784256 19119 raft_consensus.cc:374] T 2cb0b019108e49ed88a08ee7bc5bbb68 P e94d40b28f5d4842819973b4fb7b8272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:46.784361 19119 raft_consensus.cc:729] T 2cb0b019108e49ed88a08ee7bc5bbb68 P e94d40b28f5d4842819973b4fb7b8272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e94d40b28f5d4842819973b4fb7b8272, State: Initialized, Role: FOLLOWER
I20260620 23:11:46.784940 19119 consensus_queue.cc:260] T 2cb0b019108e49ed88a08ee7bc5bbb68 P e94d40b28f5d4842819973b4fb7b8272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } }
I20260620 23:11:46.786345 19119 ts_tablet_manager.cc:1428] T 2cb0b019108e49ed88a08ee7bc5bbb68 P e94d40b28f5d4842819973b4fb7b8272: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
W20260620 23:11:46.844619 18814 tablet.cc:2369] T 2cb0b019108e49ed88a08ee7bc5bbb68 P e94d40b28f5d4842819973b4fb7b8272: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:11:46.856974 18953 tablet.cc:2369] T 2cb0b019108e49ed88a08ee7bc5bbb68 P bfe128cc990a400b88234db04775deec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:11:46.872304 19092 tablet.cc:2369] T 2cb0b019108e49ed88a08ee7bc5bbb68 P 663f3caaf3a24449ae9075cd10afb8c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:11:46.998483 19122 raft_consensus.cc:482] T 2cb0b019108e49ed88a08ee7bc5bbb68 P 663f3caaf3a24449ae9075cd10afb8c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:11:46.998854 19122 raft_consensus.cc:504] T 2cb0b019108e49ed88a08ee7bc5bbb68 P 663f3caaf3a24449ae9075cd10afb8c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } }
I20260620 23:11:47.000707 19122 leader_election.cc:288] T 2cb0b019108e49ed88a08ee7bc5bbb68 P 663f3caaf3a24449ae9075cd10afb8c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bfe128cc990a400b88234db04775deec (127.10.191.194:37179), e94d40b28f5d4842819973b4fb7b8272 (127.10.191.193:37597)
I20260620 23:11:47.006997 19112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:47.001029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:52181 (local address 127.10.191.194:37179)
0620 23:11:47.001253 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:47.001259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:47.001283 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:11:47.001478 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:47.001487 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:47.001582 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:47.001682 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:11:47.002169 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.003210 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.005035 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.006073 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.006207 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:47.006258 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:47.006640 (+ 382us) server_negotiation.cc:299] Negotiation successful
0620 23:11:47.006745 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 23:11:47.007041 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:47.000888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.194:37179 (local address 127.10.191.195:52181)
0620 23:11:47.001148 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:47.001170 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:47.001296 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:47.001739 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:47.001747 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:47.001785 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:11:47.002003 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:47.002016 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.003352 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.003358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:47.004874 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:47.004886 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.006241 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.006248 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:47.006435 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:47.006471 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 23:11:47.006581 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 23:11:47.006714 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":24}
I20260620 23:11:47.007937 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:47.001272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:60943 (local address 127.10.191.193:37597)
0620 23:11:47.001514 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:47.001522 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:47.001974 (+ 452us) server_negotiation.cc:407] Connection header received
0620 23:11:47.002083 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:47.002090 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:47.002184 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:47.002259 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:11:47.002933 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.003898 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.005821 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.006973 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.007132 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:47.007201 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:47.007681 (+ 480us) server_negotiation.cc:299] Negotiation successful
0620 23:11:47.007781 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 23:11:47.008157 19124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:47.001172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:37597 (local address 127.10.191.195:60943)
0620 23:11:47.001875 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:47.001888 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:47.001983 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:47.002303 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:47.002308 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:47.002331 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:11:47.002717 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:47.002729 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.004031 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.004035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:47.005656 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:47.005669 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.007338 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.007344 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:47.007498 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:47.007520 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 23:11:47.007624 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 23:11:47.007770 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"mutex_wait_us":3,"thread_start_us":391,"threads_started":1}
I20260620 23:11:47.008229 18906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cb0b019108e49ed88a08ee7bc5bbb68" candidate_uuid: "663f3caaf3a24449ae9075cd10afb8c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfe128cc990a400b88234db04775deec" is_pre_election: true
I20260620 23:11:47.008805 18906 raft_consensus.cc:2454] T 2cb0b019108e49ed88a08ee7bc5bbb68 P bfe128cc990a400b88234db04775deec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 663f3caaf3a24449ae9075cd10afb8c7 in term 0.
I20260620 23:11:47.009202 18768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cb0b019108e49ed88a08ee7bc5bbb68" candidate_uuid: "663f3caaf3a24449ae9075cd10afb8c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e94d40b28f5d4842819973b4fb7b8272" is_pre_election: true
I20260620 23:11:47.009439 18975 leader_election.cc:302] T 2cb0b019108e49ed88a08ee7bc5bbb68 P 663f3caaf3a24449ae9075cd10afb8c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 663f3caaf3a24449ae9075cd10afb8c7, bfe128cc990a400b88234db04775deec; no voters:
I20260620 23:11:47.009732 18768 raft_consensus.cc:2454] T 2cb0b019108e49ed88a08ee7bc5bbb68 P e94d40b28f5d4842819973b4fb7b8272 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 663f3caaf3a24449ae9075cd10afb8c7 in term 0.
I20260620 23:11:47.009869 19122 raft_consensus.cc:2781] T 2cb0b019108e49ed88a08ee7bc5bbb68 P 663f3caaf3a24449ae9075cd10afb8c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:11:47.010025 19122 raft_consensus.cc:482] T 2cb0b019108e49ed88a08ee7bc5bbb68 P 663f3caaf3a24449ae9075cd10afb8c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:11:47.010141 19122 raft_consensus.cc:3037] T 2cb0b019108e49ed88a08ee7bc5bbb68 P 663f3caaf3a24449ae9075cd10afb8c7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:47.012451 19122 raft_consensus.cc:504] T 2cb0b019108e49ed88a08ee7bc5bbb68 P 663f3caaf3a24449ae9075cd10afb8c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } }
I20260620 23:11:47.013156 19122 leader_election.cc:288] T 2cb0b019108e49ed88a08ee7bc5bbb68 P 663f3caaf3a24449ae9075cd10afb8c7 [CANDIDATE]: Term 1 election: Requested vote from peers bfe128cc990a400b88234db04775deec (127.10.191.194:37179), e94d40b28f5d4842819973b4fb7b8272 (127.10.191.193:37597)
I20260620 23:11:47.013697 18768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cb0b019108e49ed88a08ee7bc5bbb68" candidate_uuid: "663f3caaf3a24449ae9075cd10afb8c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e94d40b28f5d4842819973b4fb7b8272"
I20260620 23:11:47.013906 18768 raft_consensus.cc:3037] T 2cb0b019108e49ed88a08ee7bc5bbb68 P e94d40b28f5d4842819973b4fb7b8272 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:47.013880 18906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cb0b019108e49ed88a08ee7bc5bbb68" candidate_uuid: "663f3caaf3a24449ae9075cd10afb8c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfe128cc990a400b88234db04775deec"
I20260620 23:11:47.014073 18906 raft_consensus.cc:3037] T 2cb0b019108e49ed88a08ee7bc5bbb68 P bfe128cc990a400b88234db04775deec [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:47.017123 18768 raft_consensus.cc:2454] T 2cb0b019108e49ed88a08ee7bc5bbb68 P e94d40b28f5d4842819973b4fb7b8272 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663f3caaf3a24449ae9075cd10afb8c7 in term 1.
I20260620 23:11:47.017228 18906 raft_consensus.cc:2454] T 2cb0b019108e49ed88a08ee7bc5bbb68 P bfe128cc990a400b88234db04775deec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663f3caaf3a24449ae9075cd10afb8c7 in term 1.
I20260620 23:11:47.017602 18972 leader_election.cc:302] T 2cb0b019108e49ed88a08ee7bc5bbb68 P 663f3caaf3a24449ae9075cd10afb8c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 663f3caaf3a24449ae9075cd10afb8c7, e94d40b28f5d4842819973b4fb7b8272; no voters:
I20260620 23:11:47.017933 19122 raft_consensus.cc:2781] T 2cb0b019108e49ed88a08ee7bc5bbb68 P 663f3caaf3a24449ae9075cd10afb8c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:11:47.018354 19122 raft_consensus.cc:686] T 2cb0b019108e49ed88a08ee7bc5bbb68 P 663f3caaf3a24449ae9075cd10afb8c7 [term 1 LEADER]: Becoming Leader. State: Replica: 663f3caaf3a24449ae9075cd10afb8c7, State: Running, Role: LEADER
I20260620 23:11:47.018944 19122 consensus_queue.cc:237] T 2cb0b019108e49ed88a08ee7bc5bbb68 P 663f3caaf3a24449ae9075cd10afb8c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } }
I20260620 23:11:47.023316 18605 catalog_manager.cc:5495] T 2cb0b019108e49ed88a08ee7bc5bbb68 P 663f3caaf3a24449ae9075cd10afb8c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 663f3caaf3a24449ae9075cd10afb8c7 (127.10.191.195). New cstate: current_term: 1 leader_uuid: "663f3caaf3a24449ae9075cd10afb8c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } health_report { overall_health: UNKNOWN } } }
I20260620 23:11:47.096969 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:47.091670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52010 (local address 127.10.191.195:42731)
0620 23:11:47.091830 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:47.091835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:47.091965 (+ 130us) server_negotiation.cc:407] Connection header received
0620 23:11:47.092158 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:47.092162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:47.092247 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:47.092373 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:11:47.092671 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.093364 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.094680 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.095218 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.095319 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:47.095794 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:11:47.096390 (+ 596us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:11:47.096551 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:47.096644 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 23:11:47.096721 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 23:11:47.104952 18768 raft_consensus.cc:1261] T 2cb0b019108e49ed88a08ee7bc5bbb68 P e94d40b28f5d4842819973b4fb7b8272 [term 1 FOLLOWER]: Refusing update from remote peer 663f3caaf3a24449ae9075cd10afb8c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:11:47.105115 18906 raft_consensus.cc:1261] T 2cb0b019108e49ed88a08ee7bc5bbb68 P bfe128cc990a400b88234db04775deec [term 1 FOLLOWER]: Refusing update from remote peer 663f3caaf3a24449ae9075cd10afb8c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:11:47.105933 19122 consensus_queue.cc:1035] T 2cb0b019108e49ed88a08ee7bc5bbb68 P 663f3caaf3a24449ae9075cd10afb8c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:11:47.106261 19125 consensus_queue.cc:1035] T 2cb0b019108e49ed88a08ee7bc5bbb68 P 663f3caaf3a24449ae9075cd10afb8c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:11:47.124379 19130 mvcc.cc:205] Tried to move back new op lower bound from 7299060150692311040 to 7299060150353776640. Current Snapshot: MvccSnapshot[applied={T|T < 7299060150692311040}]
I20260620 23:11:47.133460 18605 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:11:47.133740 18605 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:11:47.137336 18605 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 61bc181e1aab415b945d91121993c454: Sending DeleteTablet for 3 replicas of tablet 2cb0b019108e49ed88a08ee7bc5bbb68
I20260620 23:11:47.138461 18882 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cb0b019108e49ed88a08ee7bc5bbb68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:11:47 UTC) from {username='oryx', principal='oryx/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:55916
I20260620 23:11:47.138533 18743 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cb0b019108e49ed88a08ee7bc5bbb68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:11:47 UTC) from {username='oryx', principal='oryx/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:41416
I20260620 23:11:47.138635 19021 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cb0b019108e49ed88a08ee7bc5bbb68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:11:47 UTC) from {username='oryx', principal='oryx/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:51996
I20260620 23:11:47.139379 19138 tablet_replica.cc:330] T 2cb0b019108e49ed88a08ee7bc5bbb68 P e94d40b28f5d4842819973b4fb7b8272: stopping tablet replica
I20260620 23:11:47.139545 18606 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 23:11:47.139921 19138 raft_consensus.cc:2229] T 2cb0b019108e49ed88a08ee7bc5bbb68 P e94d40b28f5d4842819973b4fb7b8272 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260620 23:11:47.140188 18606 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:11:47.140470 19138 raft_consensus.cc:2258] T 2cb0b019108e49ed88a08ee7bc5bbb68 P e94d40b28f5d4842819973b4fb7b8272 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:11:47.145453 19138 ts_tablet_manager.cc:1905] T 2cb0b019108e49ed88a08ee7bc5bbb68 P e94d40b28f5d4842819973b4fb7b8272: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:11:47.146981 18743 tablet_service.cc:1467] Processing CreateTablet for tablet d42afbe68b8b44d98234b7f3dd846c08 (DEFAULT_TABLE table=test-table [id=ba70a2efc6334555b3bc0ce5f8d87a31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:11:47.147913 19020 tablet_service.cc:1467] Processing CreateTablet for tablet d42afbe68b8b44d98234b7f3dd846c08 (DEFAULT_TABLE table=test-table [id=ba70a2efc6334555b3bc0ce5f8d87a31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:11:47.148432 19020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d42afbe68b8b44d98234b7f3dd846c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:11:47.156186 18743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d42afbe68b8b44d98234b7f3dd846c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:11:47.159207 19116 tablet_bootstrap.cc:492] T d42afbe68b8b44d98234b7f3dd846c08 P 663f3caaf3a24449ae9075cd10afb8c7: Bootstrap starting.
I20260620 23:11:47.161396 19116 tablet_bootstrap.cc:654] T d42afbe68b8b44d98234b7f3dd846c08 P 663f3caaf3a24449ae9075cd10afb8c7: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:47.162325 18881 tablet_service.cc:1467] Processing CreateTablet for tablet d42afbe68b8b44d98234b7f3dd846c08 (DEFAULT_TABLE table=test-table [id=ba70a2efc6334555b3bc0ce5f8d87a31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:11:47.162814 19119 tablet_bootstrap.cc:492] T d42afbe68b8b44d98234b7f3dd846c08 P e94d40b28f5d4842819973b4fb7b8272: Bootstrap starting.
I20260620 23:11:47.162895 18881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d42afbe68b8b44d98234b7f3dd846c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:11:47.164345 19119 tablet_bootstrap.cc:654] T d42afbe68b8b44d98234b7f3dd846c08 P e94d40b28f5d4842819973b4fb7b8272: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:47.172655 19115 tablet_bootstrap.cc:492] T d42afbe68b8b44d98234b7f3dd846c08 P bfe128cc990a400b88234db04775deec: Bootstrap starting.
I20260620 23:11:47.173185 19137 tablet_replica.cc:330] T 2cb0b019108e49ed88a08ee7bc5bbb68 P bfe128cc990a400b88234db04775deec: stopping tablet replica
I20260620 23:11:47.173677 19119 tablet_bootstrap.cc:492] T d42afbe68b8b44d98234b7f3dd846c08 P e94d40b28f5d4842819973b4fb7b8272: No bootstrap required, opened a new log
I20260620 23:11:47.173686 19137 raft_consensus.cc:2229] T 2cb0b019108e49ed88a08ee7bc5bbb68 P bfe128cc990a400b88234db04775deec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:11:47.173841 19119 ts_tablet_manager.cc:1397] T d42afbe68b8b44d98234b7f3dd846c08 P e94d40b28f5d4842819973b4fb7b8272: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.008s
I20260620 23:11:47.174185 19137 raft_consensus.cc:2258] T 2cb0b019108e49ed88a08ee7bc5bbb68 P bfe128cc990a400b88234db04775deec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:11:47.174311 19115 tablet_bootstrap.cc:654] T d42afbe68b8b44d98234b7f3dd846c08 P bfe128cc990a400b88234db04775deec: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:47.174991 19119 raft_consensus.cc:348] T d42afbe68b8b44d98234b7f3dd846c08 P e94d40b28f5d4842819973b4fb7b8272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } }
I20260620 23:11:47.175323 19119 raft_consensus.cc:374] T d42afbe68b8b44d98234b7f3dd846c08 P e94d40b28f5d4842819973b4fb7b8272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:47.175444 19119 raft_consensus.cc:729] T d42afbe68b8b44d98234b7f3dd846c08 P e94d40b28f5d4842819973b4fb7b8272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e94d40b28f5d4842819973b4fb7b8272, State: Initialized, Role: FOLLOWER
I20260620 23:11:47.175803 19119 consensus_queue.cc:260] T d42afbe68b8b44d98234b7f3dd846c08 P e94d40b28f5d4842819973b4fb7b8272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } }
I20260620 23:11:47.176373 19119 ts_tablet_manager.cc:1428] T d42afbe68b8b44d98234b7f3dd846c08 P e94d40b28f5d4842819973b4fb7b8272: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 23:11:47.177328 19116 tablet_bootstrap.cc:492] T d42afbe68b8b44d98234b7f3dd846c08 P 663f3caaf3a24449ae9075cd10afb8c7: No bootstrap required, opened a new log
I20260620 23:11:47.177515 19116 ts_tablet_manager.cc:1397] T d42afbe68b8b44d98234b7f3dd846c08 P 663f3caaf3a24449ae9075cd10afb8c7: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.007s
I20260620 23:11:47.178233 19137 ts_tablet_manager.cc:1905] T 2cb0b019108e49ed88a08ee7bc5bbb68 P bfe128cc990a400b88234db04775deec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:11:47.178583 19116 raft_consensus.cc:348] T d42afbe68b8b44d98234b7f3dd846c08 P 663f3caaf3a24449ae9075cd10afb8c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } }
I20260620 23:11:47.178891 19116 raft_consensus.cc:374] T d42afbe68b8b44d98234b7f3dd846c08 P 663f3caaf3a24449ae9075cd10afb8c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:47.179003 19116 raft_consensus.cc:729] T d42afbe68b8b44d98234b7f3dd846c08 P 663f3caaf3a24449ae9075cd10afb8c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663f3caaf3a24449ae9075cd10afb8c7, State: Initialized, Role: FOLLOWER
I20260620 23:11:47.179311 19116 consensus_queue.cc:260] T d42afbe68b8b44d98234b7f3dd846c08 P 663f3caaf3a24449ae9075cd10afb8c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } }
I20260620 23:11:47.180138 19116 ts_tablet_manager.cc:1428] T d42afbe68b8b44d98234b7f3dd846c08 P 663f3caaf3a24449ae9075cd10afb8c7: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 23:11:47.180871 19139 tablet_replica.cc:330] T 2cb0b019108e49ed88a08ee7bc5bbb68 P 663f3caaf3a24449ae9075cd10afb8c7: stopping tablet replica
I20260620 23:11:47.181267 19138 ts_tablet_manager.cc:1918] T 2cb0b019108e49ed88a08ee7bc5bbb68 P e94d40b28f5d4842819973b4fb7b8272: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:11:47.181231 19139 raft_consensus.cc:2229] T 2cb0b019108e49ed88a08ee7bc5bbb68 P 663f3caaf3a24449ae9075cd10afb8c7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:11:47.181905 19139 raft_consensus.cc:2258] T 2cb0b019108e49ed88a08ee7bc5bbb68 P 663f3caaf3a24449ae9075cd10afb8c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:11:47.181931 19138 log.cc:1196] T 2cb0b019108e49ed88a08ee7bc5bbb68 P e94d40b28f5d4842819973b4fb7b8272: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-0/wal/wals/2cb0b019108e49ed88a08ee7bc5bbb68
I20260620 23:11:47.182477 19138 ts_tablet_manager.cc:1939] T 2cb0b019108e49ed88a08ee7bc5bbb68 P e94d40b28f5d4842819973b4fb7b8272: Deleting consensus metadata
I20260620 23:11:47.188868 19139 ts_tablet_manager.cc:1905] T 2cb0b019108e49ed88a08ee7bc5bbb68 P 663f3caaf3a24449ae9075cd10afb8c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:11:47.191480 18590 catalog_manager.cc:4841] TS e94d40b28f5d4842819973b4fb7b8272 (127.10.191.193:37597): tablet 2cb0b019108e49ed88a08ee7bc5bbb68 (table test-table [id=5a7802845c174f2694f2397cf6ef489b]) successfully deleted
I20260620 23:11:47.192610 19137 ts_tablet_manager.cc:1918] T 2cb0b019108e49ed88a08ee7bc5bbb68 P bfe128cc990a400b88234db04775deec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:11:47.192706 19115 tablet_bootstrap.cc:492] T d42afbe68b8b44d98234b7f3dd846c08 P bfe128cc990a400b88234db04775deec: No bootstrap required, opened a new log
I20260620 23:11:47.192915 19137 log.cc:1196] T 2cb0b019108e49ed88a08ee7bc5bbb68 P bfe128cc990a400b88234db04775deec: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-1/wal/wals/2cb0b019108e49ed88a08ee7bc5bbb68
I20260620 23:11:47.192935 19115 ts_tablet_manager.cc:1397] T d42afbe68b8b44d98234b7f3dd846c08 P bfe128cc990a400b88234db04775deec: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.008s
I20260620 23:11:47.193518 19137 ts_tablet_manager.cc:1939] T 2cb0b019108e49ed88a08ee7bc5bbb68 P bfe128cc990a400b88234db04775deec: Deleting consensus metadata
I20260620 23:11:47.193882 19115 raft_consensus.cc:348] T d42afbe68b8b44d98234b7f3dd846c08 P bfe128cc990a400b88234db04775deec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } }
I20260620 23:11:47.194116 19115 raft_consensus.cc:374] T d42afbe68b8b44d98234b7f3dd846c08 P bfe128cc990a400b88234db04775deec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:47.194229 19115 raft_consensus.cc:729] T d42afbe68b8b44d98234b7f3dd846c08 P bfe128cc990a400b88234db04775deec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfe128cc990a400b88234db04775deec, State: Initialized, Role: FOLLOWER
I20260620 23:11:47.194506 19139 ts_tablet_manager.cc:1918] T 2cb0b019108e49ed88a08ee7bc5bbb68 P 663f3caaf3a24449ae9075cd10afb8c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:11:47.194685 19139 log.cc:1196] T 2cb0b019108e49ed88a08ee7bc5bbb68 P 663f3caaf3a24449ae9075cd10afb8c7: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-2/wal/wals/2cb0b019108e49ed88a08ee7bc5bbb68
I20260620 23:11:47.194545 19115 consensus_queue.cc:260] T d42afbe68b8b44d98234b7f3dd846c08 P bfe128cc990a400b88234db04775deec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } }
I20260620 23:11:47.195217 19115 ts_tablet_manager.cc:1428] T d42afbe68b8b44d98234b7f3dd846c08 P bfe128cc990a400b88234db04775deec: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 23:11:47.195360 18593 catalog_manager.cc:4841] TS bfe128cc990a400b88234db04775deec (127.10.191.194:37179): tablet 2cb0b019108e49ed88a08ee7bc5bbb68 (table test-table [id=5a7802845c174f2694f2397cf6ef489b]) successfully deleted
I20260620 23:11:47.199280 19139 ts_tablet_manager.cc:1939] T 2cb0b019108e49ed88a08ee7bc5bbb68 P 663f3caaf3a24449ae9075cd10afb8c7: Deleting consensus metadata
I20260620 23:11:47.201030 18590 catalog_manager.cc:4841] TS 663f3caaf3a24449ae9075cd10afb8c7 (127.10.191.195:42731): tablet 2cb0b019108e49ed88a08ee7bc5bbb68 (table test-table [id=5a7802845c174f2694f2397cf6ef489b]) successfully deleted
I20260620 23:11:47.307960 19123 raft_consensus.cc:482] T d42afbe68b8b44d98234b7f3dd846c08 P e94d40b28f5d4842819973b4fb7b8272 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:11:47.308225 19123 raft_consensus.cc:504] T d42afbe68b8b44d98234b7f3dd846c08 P e94d40b28f5d4842819973b4fb7b8272 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } }
I20260620 23:11:47.309800 19123 leader_election.cc:288] T d42afbe68b8b44d98234b7f3dd846c08 P e94d40b28f5d4842819973b4fb7b8272 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bfe128cc990a400b88234db04775deec (127.10.191.194:37179), 663f3caaf3a24449ae9075cd10afb8c7 (127.10.191.195:42731)
I20260620 23:11:47.319170 19112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:47.310857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:34407 (local address 127.10.191.194:37179)
0620 23:11:47.311120 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:47.311129 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:47.311747 (+ 618us) server_negotiation.cc:407] Connection header received
0620 23:11:47.312055 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:47.312061 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:47.312151 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:47.312257 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:11:47.313261 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.314233 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.317405 (+ 3172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.318166 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.318348 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:47.318422 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:47.318740 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 23:11:47.318882 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 23:11:47.319293 19144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:47.310042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.194:37179 (local address 127.10.191.193:34407)
0620 23:11:47.311624 (+ 1582us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:47.311645 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:47.311878 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:47.312286 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:47.312291 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:47.312316 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:11:47.313091 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:47.313104 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.314352 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.314360 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:47.317182 (+ 2822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:47.317193 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.318373 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.318379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:47.318544 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:47.318567 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 23:11:47.318714 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 23:11:47.318842 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1420,"mutex_wait_us":207,"thread_start_us":1587,"threads_started":1}
I20260620 23:11:47.320046 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:47.310535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:46675 (local address 127.10.191.195:42731)
0620 23:11:47.310716 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:47.310722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:47.311988 (+ 1266us) server_negotiation.cc:407] Connection header received
0620 23:11:47.312209 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:47.312229 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:47.312310 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:47.312453 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:11:47.313147 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.314333 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.317353 (+ 3020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.318720 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.318924 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:47.318998 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:47.319648 (+ 650us) server_negotiation.cc:299] Negotiation successful
0620 23:11:47.319787 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 23:11:47.320109 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:47.310172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.195:42731 (local address 127.10.191.193:46675)
0620 23:11:47.311907 (+ 1735us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:47.311923 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:47.312069 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:47.312441 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:47.312447 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:47.312486 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:11:47.312911 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:47.312924 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.314475 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.314482 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:47.317180 (+ 2698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:47.317193 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.319172 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.319179 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:47.319376 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:47.319456 (+ 80us) client_negotiation.cc:769] Sending connection context
0620 23:11:47.319588 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 23:11:47.319718 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1622,"mutex_wait_us":178,"thread_start_us":1129,"threads_started":1}
I20260620 23:11:47.320151 18906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d42afbe68b8b44d98234b7f3dd846c08" candidate_uuid: "e94d40b28f5d4842819973b4fb7b8272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfe128cc990a400b88234db04775deec" is_pre_election: true
I20260620 23:11:47.320407 18906 raft_consensus.cc:2454] T d42afbe68b8b44d98234b7f3dd846c08 P bfe128cc990a400b88234db04775deec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e94d40b28f5d4842819973b4fb7b8272 in term 0.
I20260620 23:11:47.320931 18696 leader_election.cc:302] T d42afbe68b8b44d98234b7f3dd846c08 P e94d40b28f5d4842819973b4fb7b8272 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bfe128cc990a400b88234db04775deec, e94d40b28f5d4842819973b4fb7b8272; no voters:
I20260620 23:11:47.321014 19043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d42afbe68b8b44d98234b7f3dd846c08" candidate_uuid: "e94d40b28f5d4842819973b4fb7b8272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663f3caaf3a24449ae9075cd10afb8c7" is_pre_election: true
I20260620 23:11:47.321297 19123 raft_consensus.cc:2781] T d42afbe68b8b44d98234b7f3dd846c08 P e94d40b28f5d4842819973b4fb7b8272 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:11:47.321405 19123 raft_consensus.cc:482] T d42afbe68b8b44d98234b7f3dd846c08 P e94d40b28f5d4842819973b4fb7b8272 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:11:47.321476 19123 raft_consensus.cc:3037] T d42afbe68b8b44d98234b7f3dd846c08 P e94d40b28f5d4842819973b4fb7b8272 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:47.321457 19043 raft_consensus.cc:2454] T d42afbe68b8b44d98234b7f3dd846c08 P 663f3caaf3a24449ae9075cd10afb8c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e94d40b28f5d4842819973b4fb7b8272 in term 0.
I20260620 23:11:47.323865 19123 raft_consensus.cc:504] T d42afbe68b8b44d98234b7f3dd846c08 P e94d40b28f5d4842819973b4fb7b8272 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } }
I20260620 23:11:47.324491 19123 leader_election.cc:288] T d42afbe68b8b44d98234b7f3dd846c08 P e94d40b28f5d4842819973b4fb7b8272 [CANDIDATE]: Term 1 election: Requested vote from peers bfe128cc990a400b88234db04775deec (127.10.191.194:37179), 663f3caaf3a24449ae9075cd10afb8c7 (127.10.191.195:42731)
I20260620 23:11:47.324903 18906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d42afbe68b8b44d98234b7f3dd846c08" candidate_uuid: "e94d40b28f5d4842819973b4fb7b8272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfe128cc990a400b88234db04775deec"
I20260620 23:11:47.325030 19043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d42afbe68b8b44d98234b7f3dd846c08" candidate_uuid: "e94d40b28f5d4842819973b4fb7b8272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663f3caaf3a24449ae9075cd10afb8c7"
I20260620 23:11:47.325107 18906 raft_consensus.cc:3037] T d42afbe68b8b44d98234b7f3dd846c08 P bfe128cc990a400b88234db04775deec [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:47.325232 19043 raft_consensus.cc:3037] T d42afbe68b8b44d98234b7f3dd846c08 P 663f3caaf3a24449ae9075cd10afb8c7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:47.327993 18906 raft_consensus.cc:2454] T d42afbe68b8b44d98234b7f3dd846c08 P bfe128cc990a400b88234db04775deec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e94d40b28f5d4842819973b4fb7b8272 in term 1.
I20260620 23:11:47.328337 18696 leader_election.cc:302] T d42afbe68b8b44d98234b7f3dd846c08 P e94d40b28f5d4842819973b4fb7b8272 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bfe128cc990a400b88234db04775deec, e94d40b28f5d4842819973b4fb7b8272; no voters:
I20260620 23:11:47.328480 19043 raft_consensus.cc:2454] T d42afbe68b8b44d98234b7f3dd846c08 P 663f3caaf3a24449ae9075cd10afb8c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e94d40b28f5d4842819973b4fb7b8272 in term 1.
I20260620 23:11:47.328624 19123 raft_consensus.cc:2781] T d42afbe68b8b44d98234b7f3dd846c08 P e94d40b28f5d4842819973b4fb7b8272 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:11:47.331110 19123 raft_consensus.cc:686] T d42afbe68b8b44d98234b7f3dd846c08 P e94d40b28f5d4842819973b4fb7b8272 [term 1 LEADER]: Becoming Leader. State: Replica: e94d40b28f5d4842819973b4fb7b8272, State: Running, Role: LEADER
I20260620 23:11:47.331684 19123 consensus_queue.cc:237] T d42afbe68b8b44d98234b7f3dd846c08 P e94d40b28f5d4842819973b4fb7b8272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } }
I20260620 23:11:47.334952 18605 catalog_manager.cc:5495] T d42afbe68b8b44d98234b7f3dd846c08 P e94d40b28f5d4842819973b4fb7b8272 reported cstate change: term changed from 0 to 1, leader changed from <none> to e94d40b28f5d4842819973b4fb7b8272 (127.10.191.193). New cstate: current_term: 1 leader_uuid: "e94d40b28f5d4842819973b4fb7b8272" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } health_report { overall_health: UNKNOWN } } }
Jun 20 23:11:47 dist-test-slave-jkbq krb5kdc[18561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997104, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.191.254@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM
I20260620 23:11:47.400769 19153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:47.385185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:36897 (local address 127.0.0.1:58004)
0620 23:11:47.386367 (+ 1182us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:47.386388 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:47.386545 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:47.386964 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:47.386969 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:47.387937 (+ 968us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:47.388279 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:47.388292 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.389453 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.389459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:47.390990 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:47.391004 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.391683 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.391690 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:47.391865 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:47.392577 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:47.392601 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:47.395207 (+ 2606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:47.398214 (+ 3007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:47.398227 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:47.398234 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:47.398609 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:47.399041 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:47.399047 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:47.399051 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:47.399191 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:47.399711 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:47.399718 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:47.399869 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 23:11:47.400026 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 23:11:47.400307 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1059,"thread_start_us":930,"threads_started":1}
I20260620 23:11:47.400815 19152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:47.385214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58004 (local address 127.10.191.254:36897)
0620 23:11:47.386011 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:47.386018 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:47.386485 (+ 467us) server_negotiation.cc:407] Connection header received
0620 23:11:47.386702 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:47.386705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:47.386777 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:47.386938 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:47.388457 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.389315 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.391128 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.391513 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.391605 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:47.395399 (+ 3794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:47.395404 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:47.395407 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:47.395432 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:47.398090 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:47.398725 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:47.398730 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:47.398740 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:47.398925 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:47.399306 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:47.399311 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:47.399314 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:47.399565 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:47.399724 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:47.400109 (+ 385us) server_negotiation.cc:299] Negotiation successful
0620 23:11:47.400434 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":139,"server-negotiator.queue_time_us":657,"thread_start_us":470,"threads_started":1}
I20260620 23:11:47.405544 18605 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:58004:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 23:11:47.406607 18605 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:11:47.413252 18743 tablet_service.cc:1467] Processing CreateTablet for tablet b90479566c59488e8a300d0e69dad55b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cfba493b92a548dd9123faf02bad7ced]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:11:47.413364 18882 tablet_service.cc:1467] Processing CreateTablet for tablet b90479566c59488e8a300d0e69dad55b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cfba493b92a548dd9123faf02bad7ced]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:11:47.413859 18743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b90479566c59488e8a300d0e69dad55b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:11:47.413859 18882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b90479566c59488e8a300d0e69dad55b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:11:47.413858 19021 tablet_service.cc:1467] Processing CreateTablet for tablet b90479566c59488e8a300d0e69dad55b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cfba493b92a548dd9123faf02bad7ced]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:11:47.414417 19021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b90479566c59488e8a300d0e69dad55b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:11:47.421077 19115 tablet_bootstrap.cc:492] T b90479566c59488e8a300d0e69dad55b P bfe128cc990a400b88234db04775deec: Bootstrap starting.
I20260620 23:11:47.422454 19116 tablet_bootstrap.cc:492] T b90479566c59488e8a300d0e69dad55b P 663f3caaf3a24449ae9075cd10afb8c7: Bootstrap starting.
I20260620 23:11:47.422451 19115 tablet_bootstrap.cc:654] T b90479566c59488e8a300d0e69dad55b P bfe128cc990a400b88234db04775deec: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:47.424281 19116 tablet_bootstrap.cc:654] T b90479566c59488e8a300d0e69dad55b P 663f3caaf3a24449ae9075cd10afb8c7: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:47.424846 19119 tablet_bootstrap.cc:492] T b90479566c59488e8a300d0e69dad55b P e94d40b28f5d4842819973b4fb7b8272: Bootstrap starting.
I20260620 23:11:47.426405 19116 tablet_bootstrap.cc:492] T b90479566c59488e8a300d0e69dad55b P 663f3caaf3a24449ae9075cd10afb8c7: No bootstrap required, opened a new log
I20260620 23:11:47.426471 19119 tablet_bootstrap.cc:654] T b90479566c59488e8a300d0e69dad55b P e94d40b28f5d4842819973b4fb7b8272: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:47.426569 19116 ts_tablet_manager.cc:1397] T b90479566c59488e8a300d0e69dad55b P 663f3caaf3a24449ae9075cd10afb8c7: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 23:11:47.427409 19116 raft_consensus.cc:348] T b90479566c59488e8a300d0e69dad55b P 663f3caaf3a24449ae9075cd10afb8c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } }
I20260620 23:11:47.427637 19116 raft_consensus.cc:374] T b90479566c59488e8a300d0e69dad55b P 663f3caaf3a24449ae9075cd10afb8c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:47.427748 19116 raft_consensus.cc:729] T b90479566c59488e8a300d0e69dad55b P 663f3caaf3a24449ae9075cd10afb8c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663f3caaf3a24449ae9075cd10afb8c7, State: Initialized, Role: FOLLOWER
I20260620 23:11:47.428062 19115 tablet_bootstrap.cc:492] T b90479566c59488e8a300d0e69dad55b P bfe128cc990a400b88234db04775deec: No bootstrap required, opened a new log
I20260620 23:11:47.428017 19116 consensus_queue.cc:260] T b90479566c59488e8a300d0e69dad55b P 663f3caaf3a24449ae9075cd10afb8c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } }
I20260620 23:11:47.428244 19115 ts_tablet_manager.cc:1397] T b90479566c59488e8a300d0e69dad55b P bfe128cc990a400b88234db04775deec: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260620 23:11:47.428642 19116 ts_tablet_manager.cc:1428] T b90479566c59488e8a300d0e69dad55b P 663f3caaf3a24449ae9075cd10afb8c7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:11:47.428478 19122 tablet_replica.cc:439] T b90479566c59488e8a300d0e69dad55b P 663f3caaf3a24449ae9075cd10afb8c7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } } }
I20260620 23:11:47.428781 19122 tablet_replica.cc:442] T b90479566c59488e8a300d0e69dad55b P 663f3caaf3a24449ae9075cd10afb8c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:11:47.428983 19115 raft_consensus.cc:348] T b90479566c59488e8a300d0e69dad55b P bfe128cc990a400b88234db04775deec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } }
I20260620 23:11:47.429220 19115 raft_consensus.cc:374] T b90479566c59488e8a300d0e69dad55b P bfe128cc990a400b88234db04775deec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:47.429299 19115 raft_consensus.cc:729] T b90479566c59488e8a300d0e69dad55b P bfe128cc990a400b88234db04775deec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfe128cc990a400b88234db04775deec, State: Initialized, Role: FOLLOWER
I20260620 23:11:47.429575 19115 consensus_queue.cc:260] T b90479566c59488e8a300d0e69dad55b P bfe128cc990a400b88234db04775deec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } }
I20260620 23:11:47.430012 19121 tablet_replica.cc:439] T b90479566c59488e8a300d0e69dad55b P bfe128cc990a400b88234db04775deec: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } } }
I20260620 23:11:47.430382 19121 tablet_replica.cc:442] T b90479566c59488e8a300d0e69dad55b P bfe128cc990a400b88234db04775deec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:11:47.431882 19115 ts_tablet_manager.cc:1428] T b90479566c59488e8a300d0e69dad55b P bfe128cc990a400b88234db04775deec: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260620 23:11:47.434707 19119 tablet_bootstrap.cc:492] T b90479566c59488e8a300d0e69dad55b P e94d40b28f5d4842819973b4fb7b8272: No bootstrap required, opened a new log
I20260620 23:11:47.434921 19119 ts_tablet_manager.cc:1397] T b90479566c59488e8a300d0e69dad55b P e94d40b28f5d4842819973b4fb7b8272: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.003s
I20260620 23:11:47.435702 19119 raft_consensus.cc:348] T b90479566c59488e8a300d0e69dad55b P e94d40b28f5d4842819973b4fb7b8272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } }
I20260620 23:11:47.435950 19119 raft_consensus.cc:374] T b90479566c59488e8a300d0e69dad55b P e94d40b28f5d4842819973b4fb7b8272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:47.436033 19119 raft_consensus.cc:729] T b90479566c59488e8a300d0e69dad55b P e94d40b28f5d4842819973b4fb7b8272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e94d40b28f5d4842819973b4fb7b8272, State: Initialized, Role: FOLLOWER
I20260620 23:11:47.436271 19119 consensus_queue.cc:260] T b90479566c59488e8a300d0e69dad55b P e94d40b28f5d4842819973b4fb7b8272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } }
I20260620 23:11:47.436877 19119 ts_tablet_manager.cc:1428] T b90479566c59488e8a300d0e69dad55b P e94d40b28f5d4842819973b4fb7b8272: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 23:11:47.436753 19123 tablet_replica.cc:439] T b90479566c59488e8a300d0e69dad55b P e94d40b28f5d4842819973b4fb7b8272: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } } }
I20260620 23:11:47.437075 19123 tablet_replica.cc:442] T b90479566c59488e8a300d0e69dad55b P e94d40b28f5d4842819973b4fb7b8272: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:11:47.655470 19121 raft_consensus.cc:482] T b90479566c59488e8a300d0e69dad55b P bfe128cc990a400b88234db04775deec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:11:47.655748 19121 raft_consensus.cc:504] T b90479566c59488e8a300d0e69dad55b P bfe128cc990a400b88234db04775deec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } }
I20260620 23:11:47.657259 19121 leader_election.cc:288] T b90479566c59488e8a300d0e69dad55b P bfe128cc990a400b88234db04775deec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e94d40b28f5d4842819973b4fb7b8272 (127.10.191.193:37597), 663f3caaf3a24449ae9075cd10afb8c7 (127.10.191.195:42731)
I20260620 23:11:47.664099 19158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:47.657626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:42869 (local address 127.10.191.193:37597)
0620 23:11:47.658054 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:47.658058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:47.658079 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:11:47.658504 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:47.658512 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:47.658602 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:47.658745 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:11:47.659292 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.660018 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.662248 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.663113 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.663304 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:47.663381 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:47.663692 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 23:11:47.663821 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":243,"threads_started":1}
I20260620 23:11:47.664201 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:47.658336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:59055 (local address 127.10.191.195:42731)
0620 23:11:47.658493 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:47.658499 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:47.658798 (+ 299us) server_negotiation.cc:407] Connection header received
0620 23:11:47.659013 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:47.659018 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:47.659092 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:47.659186 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:11:47.659685 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.660346 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.662412 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.663247 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.663424 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:47.663500 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:47.663808 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 23:11:47.663936 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 23:11:47.664575 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:47.658204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.195:42731 (local address 127.10.191.194:59055)
0620 23:11:47.658690 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:47.658703 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:47.658837 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:47.659199 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:47.659203 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:47.659224 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:11:47.659530 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:47.659538 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.660497 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.660502 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:47.662258 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:47.662269 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.663465 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.663472 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:47.663660 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:47.663683 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 23:11:47.663785 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 23:11:47.663919 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":24,"thread_start_us":235,"threads_started":1}
I20260620 23:11:47.664551 19157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:47.657514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:37597 (local address 127.10.191.194:42869)
0620 23:11:47.657950 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:47.657976 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:47.658141 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:47.658768 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:47.658773 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:47.658802 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:11:47.659149 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:47.659157 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.660155 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.660164 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:47.662081 (+ 1917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:47.662092 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.663304 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.663310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:47.663518 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:47.663542 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 23:11:47.663658 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 23:11:47.663790 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":100,"thread_start_us":237,"threads_started":1}
I20260620 23:11:47.665288 19043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b90479566c59488e8a300d0e69dad55b" candidate_uuid: "bfe128cc990a400b88234db04775deec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663f3caaf3a24449ae9075cd10afb8c7" is_pre_election: true
I20260620 23:11:47.665315 18768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b90479566c59488e8a300d0e69dad55b" candidate_uuid: "bfe128cc990a400b88234db04775deec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e94d40b28f5d4842819973b4fb7b8272" is_pre_election: true
I20260620 23:11:47.665529 19043 raft_consensus.cc:2454] T b90479566c59488e8a300d0e69dad55b P 663f3caaf3a24449ae9075cd10afb8c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfe128cc990a400b88234db04775deec in term 0.
I20260620 23:11:47.665643 18768 raft_consensus.cc:2454] T b90479566c59488e8a300d0e69dad55b P e94d40b28f5d4842819973b4fb7b8272 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfe128cc990a400b88234db04775deec in term 0.
I20260620 23:11:47.666060 18833 leader_election.cc:302] T b90479566c59488e8a300d0e69dad55b P bfe128cc990a400b88234db04775deec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 663f3caaf3a24449ae9075cd10afb8c7, bfe128cc990a400b88234db04775deec; no voters:
I20260620 23:11:47.666428 19121 raft_consensus.cc:2781] T b90479566c59488e8a300d0e69dad55b P bfe128cc990a400b88234db04775deec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:11:47.666550 19121 raft_consensus.cc:482] T b90479566c59488e8a300d0e69dad55b P bfe128cc990a400b88234db04775deec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:11:47.666666 19121 raft_consensus.cc:3037] T b90479566c59488e8a300d0e69dad55b P bfe128cc990a400b88234db04775deec [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:47.668803 19121 raft_consensus.cc:504] T b90479566c59488e8a300d0e69dad55b P bfe128cc990a400b88234db04775deec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } }
I20260620 23:11:47.669339 19121 leader_election.cc:288] T b90479566c59488e8a300d0e69dad55b P bfe128cc990a400b88234db04775deec [CANDIDATE]: Term 1 election: Requested vote from peers e94d40b28f5d4842819973b4fb7b8272 (127.10.191.193:37597), 663f3caaf3a24449ae9075cd10afb8c7 (127.10.191.195:42731)
I20260620 23:11:47.669823 19043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b90479566c59488e8a300d0e69dad55b" candidate_uuid: "bfe128cc990a400b88234db04775deec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663f3caaf3a24449ae9075cd10afb8c7"
I20260620 23:11:47.669840 18768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b90479566c59488e8a300d0e69dad55b" candidate_uuid: "bfe128cc990a400b88234db04775deec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e94d40b28f5d4842819973b4fb7b8272"
I20260620 23:11:47.670015 19043 raft_consensus.cc:3037] T b90479566c59488e8a300d0e69dad55b P 663f3caaf3a24449ae9075cd10afb8c7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:47.670064 18768 raft_consensus.cc:3037] T b90479566c59488e8a300d0e69dad55b P e94d40b28f5d4842819973b4fb7b8272 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:47.671957 19043 raft_consensus.cc:2454] T b90479566c59488e8a300d0e69dad55b P 663f3caaf3a24449ae9075cd10afb8c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfe128cc990a400b88234db04775deec in term 1.
I20260620 23:11:47.672086 18768 raft_consensus.cc:2454] T b90479566c59488e8a300d0e69dad55b P e94d40b28f5d4842819973b4fb7b8272 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfe128cc990a400b88234db04775deec in term 1.
I20260620 23:11:47.672287 18833 leader_election.cc:302] T b90479566c59488e8a300d0e69dad55b P bfe128cc990a400b88234db04775deec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 663f3caaf3a24449ae9075cd10afb8c7, bfe128cc990a400b88234db04775deec; no voters:
I20260620 23:11:47.672581 19121 raft_consensus.cc:2781] T b90479566c59488e8a300d0e69dad55b P bfe128cc990a400b88234db04775deec [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:11:47.673074 19121 raft_consensus.cc:686] T b90479566c59488e8a300d0e69dad55b P bfe128cc990a400b88234db04775deec [term 1 LEADER]: Becoming Leader. State: Replica: bfe128cc990a400b88234db04775deec, State: Running, Role: LEADER
I20260620 23:11:47.673456 19121 consensus_queue.cc:237] T b90479566c59488e8a300d0e69dad55b P bfe128cc990a400b88234db04775deec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } }
I20260620 23:11:47.675195 19160 tablet_replica.cc:439] T b90479566c59488e8a300d0e69dad55b P bfe128cc990a400b88234db04775deec: TxnStatusTablet state changed. Reason: New leader bfe128cc990a400b88234db04775deec. Latest consensus state: current_term: 1 leader_uuid: "bfe128cc990a400b88234db04775deec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } } }
I20260620 23:11:47.675607 19160 tablet_replica.cc:442] T b90479566c59488e8a300d0e69dad55b P bfe128cc990a400b88234db04775deec: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:11:47.676316 19162 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 23:11:47.676528 19162 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 23:11:47.677070 18604 catalog_manager.cc:5495] T b90479566c59488e8a300d0e69dad55b P bfe128cc990a400b88234db04775deec reported cstate change: term changed from 0 to 1, leader changed from <none> to bfe128cc990a400b88234db04775deec (127.10.191.194). New cstate: current_term: 1 leader_uuid: "bfe128cc990a400b88234db04775deec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } health_report { overall_health: UNKNOWN } } }
I20260620 23:11:47.731843 19112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:47.726401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55920 (local address 127.10.191.194:37179)
0620 23:11:47.726616 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:47.726622 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:47.726641 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:11:47.726900 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:47.726905 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:47.727004 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:47.727146 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:11:47.727487 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.728274 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.729565 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.730000 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.730166 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:47.730604 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:11:47.731172 (+ 568us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:11:47.731320 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:47.731415 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 23:11:47.731494 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 23:11:47.731838 19153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:47.726266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.194:37179 (local address 127.0.0.1:55920)
0620 23:11:47.726494 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:47.726516 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:47.726652 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:47.727169 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:47.727173 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:47.727195 (+ 22us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:11:47.727358 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:47.727367 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.728423 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.728435 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:47.729424 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:47.729435 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.730190 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.730196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:47.730351 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:47.730401 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:11:47.731283 (+ 882us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:11:47.731302 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 23:11:47.731390 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 23:11:47.731534 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 23:11:47.735929 18768 raft_consensus.cc:1261] T b90479566c59488e8a300d0e69dad55b P e94d40b28f5d4842819973b4fb7b8272 [term 1 FOLLOWER]: Refusing update from remote peer bfe128cc990a400b88234db04775deec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:11:47.736258 19043 raft_consensus.cc:1261] T b90479566c59488e8a300d0e69dad55b P 663f3caaf3a24449ae9075cd10afb8c7 [term 1 FOLLOWER]: Refusing update from remote peer bfe128cc990a400b88234db04775deec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:11:47.736563 19121 consensus_queue.cc:1035] T b90479566c59488e8a300d0e69dad55b P bfe128cc990a400b88234db04775deec [LEADER]: Connected to new peer: Peer: permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:11:47.737185 19121 consensus_queue.cc:1035] T b90479566c59488e8a300d0e69dad55b P bfe128cc990a400b88234db04775deec [LEADER]: Connected to new peer: Peer: permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:11:47.748607 19122 tablet_replica.cc:439] T b90479566c59488e8a300d0e69dad55b P 663f3caaf3a24449ae9075cd10afb8c7: TxnStatusTablet state changed. Reason: New leader bfe128cc990a400b88234db04775deec. Latest consensus state: current_term: 1 leader_uuid: "bfe128cc990a400b88234db04775deec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } } }
I20260620 23:11:47.748968 19122 tablet_replica.cc:442] T b90479566c59488e8a300d0e69dad55b P 663f3caaf3a24449ae9075cd10afb8c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:11:47.750110 19123 tablet_replica.cc:439] T b90479566c59488e8a300d0e69dad55b P e94d40b28f5d4842819973b4fb7b8272: TxnStatusTablet state changed. Reason: New leader bfe128cc990a400b88234db04775deec. Latest consensus state: current_term: 1 leader_uuid: "bfe128cc990a400b88234db04775deec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } } }
I20260620 23:11:47.750391 19123 tablet_replica.cc:442] T b90479566c59488e8a300d0e69dad55b P e94d40b28f5d4842819973b4fb7b8272: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:11:47.751034 19165 mvcc.cc:205] Tried to move back new op lower bound from 7299060153279463424 to 7299060153033670656. Current Snapshot: MvccSnapshot[applied={T|T < 7299060153279463424}]
I20260620 23:11:47.752038 19121 tablet_replica.cc:439] T b90479566c59488e8a300d0e69dad55b P bfe128cc990a400b88234db04775deec: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bfe128cc990a400b88234db04775deec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } } }
I20260620 23:11:47.752369 19121 tablet_replica.cc:442] T b90479566c59488e8a300d0e69dad55b P bfe128cc990a400b88234db04775deec: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:11:47.756384 19168 mvcc.cc:205] Tried to move back new op lower bound from 7299060153279463424 to 7299060153033670656. Current Snapshot: MvccSnapshot[applied={T|T < 7299060153279463424}]
I20260620 23:11:47.758008 19160 tablet_replica.cc:439] T b90479566c59488e8a300d0e69dad55b P bfe128cc990a400b88234db04775deec: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bfe128cc990a400b88234db04775deec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } } }
I20260620 23:11:47.758234 19160 tablet_replica.cc:442] T b90479566c59488e8a300d0e69dad55b P bfe128cc990a400b88234db04775deec: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:11:47.759871 19123 tablet_replica.cc:439] T b90479566c59488e8a300d0e69dad55b P e94d40b28f5d4842819973b4fb7b8272: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bfe128cc990a400b88234db04775deec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } } }
I20260620 23:11:47.760179 19123 tablet_replica.cc:442] T b90479566c59488e8a300d0e69dad55b P e94d40b28f5d4842819973b4fb7b8272: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:11:47.760967 19122 tablet_replica.cc:439] T b90479566c59488e8a300d0e69dad55b P 663f3caaf3a24449ae9075cd10afb8c7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bfe128cc990a400b88234db04775deec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } } }
I20260620 23:11:47.761194 19122 tablet_replica.cc:442] T b90479566c59488e8a300d0e69dad55b P 663f3caaf3a24449ae9075cd10afb8c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:11:47.769912 19158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:47.763102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41430 (local address 127.10.191.193:37597)
0620 23:11:47.763286 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:47.763292 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:47.763868 (+ 576us) server_negotiation.cc:407] Connection header received
0620 23:11:47.764049 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:47.764054 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:47.764169 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:47.764328 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:11:47.764753 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.765864 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.767415 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.768069 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.768194 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:47.768570 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:11:47.769271 (+ 701us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:11:47.769410 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:47.769534 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 23:11:47.769621 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 23:11:47.785882 19112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:47.779991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:37717 (local address 127.10.191.194:37179)
0620 23:11:47.780172 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:47.780177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:47.780323 (+ 146us) server_negotiation.cc:407] Connection header received
0620 23:11:47.780540 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:47.780547 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:47.780627 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:47.780729 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:11:47.781297 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.782151 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.784164 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.784662 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.784760 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:47.785261 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:11:47.785369 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:11:47.785431 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:47.785577 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 23:11:47.785663 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 23:11:47.786072 19178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:47.779636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.194:37179 (local address 127.10.191.193:37717)
0620 23:11:47.780223 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:47.780247 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:47.780377 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:47.780765 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:47.780770 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:47.780783 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:11:47.781134 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:47.781145 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.782302 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.782307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:47.784023 (+ 1716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:47.784038 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.784828 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.784835 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:47.785014 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:47.785079 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:11:47.785461 (+ 382us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:11:47.785472 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 23:11:47.785554 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 23:11:47.785707 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":274,"threads_started":1}
I20260620 23:11:47.795336 19043 raft_consensus.cc:1261] T d42afbe68b8b44d98234b7f3dd846c08 P 663f3caaf3a24449ae9075cd10afb8c7 [term 1 FOLLOWER]: Refusing update from remote peer e94d40b28f5d4842819973b4fb7b8272: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:11:47.795492 18906 raft_consensus.cc:1261] T d42afbe68b8b44d98234b7f3dd846c08 P bfe128cc990a400b88234db04775deec [term 1 FOLLOWER]: Refusing update from remote peer e94d40b28f5d4842819973b4fb7b8272: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:11:47.796162 19145 consensus_queue.cc:1035] T d42afbe68b8b44d98234b7f3dd846c08 P e94d40b28f5d4842819973b4fb7b8272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:11:47.796787 19145 consensus_queue.cc:1035] T d42afbe68b8b44d98234b7f3dd846c08 P e94d40b28f5d4842819973b4fb7b8272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:11:47.836381 19158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:47.827207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:55317 (local address 127.10.191.193:37597)
0620 23:11:47.827360 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:47.827367 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:47.829553 (+ 2186us) server_negotiation.cc:407] Connection header received
0620 23:11:47.829692 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:47.829697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:47.829774 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:47.829856 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:11:47.830571 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.831740 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.833992 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.834713 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.834813 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:47.835624 (+ 811us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:11:47.835758 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:11:47.835848 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:47.836028 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 23:11:47.836104 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 23:11:47.836573 19184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:47.826782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:37597 (local address 127.10.191.194:55317)
0620 23:11:47.829452 (+ 2670us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:47.829468 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:47.829574 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:47.830009 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:47.830015 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:47.830028 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:11:47.830388 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:47.830399 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.831911 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.831916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:47.833820 (+ 1904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:47.833836 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.835146 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.835153 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:47.835338 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:47.835404 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:11:47.835904 (+ 500us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:11:47.835917 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 23:11:47.836018 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 23:11:47.836165 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2517,"thread_start_us":2386,"threads_started":1}
I20260620 23:11:47.873215 18606 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:11:47.873489 18606 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:11:47.877046 18606 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 61bc181e1aab415b945d91121993c454: Sending DeleteTablet for 3 replicas of tablet d42afbe68b8b44d98234b7f3dd846c08
I20260620 23:11:47.877902 19021 tablet_service.cc:1514] Processing DeleteTablet for tablet d42afbe68b8b44d98234b7f3dd846c08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:11:47 UTC) from {username='oryx', principal='oryx/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:51996
I20260620 23:11:47.878588 19187 tablet_replica.cc:330] T d42afbe68b8b44d98234b7f3dd846c08 P 663f3caaf3a24449ae9075cd10afb8c7: stopping tablet replica
I20260620 23:11:47.878876 19187 raft_consensus.cc:2229] T d42afbe68b8b44d98234b7f3dd846c08 P 663f3caaf3a24449ae9075cd10afb8c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:11:47.879047 18743 tablet_service.cc:1514] Processing DeleteTablet for tablet d42afbe68b8b44d98234b7f3dd846c08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:11:47 UTC) from {username='oryx', principal='oryx/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:41416
I20260620 23:11:47.879196 19187 raft_consensus.cc:2258] T d42afbe68b8b44d98234b7f3dd846c08 P 663f3caaf3a24449ae9075cd10afb8c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:11:47.880138 19187 ts_tablet_manager.cc:1905] T d42afbe68b8b44d98234b7f3dd846c08 P 663f3caaf3a24449ae9075cd10afb8c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:11:47.883531 18882 tablet_service.cc:1514] Processing DeleteTablet for tablet d42afbe68b8b44d98234b7f3dd846c08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:11:47 UTC) from {username='oryx', principal='oryx/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:55916
I20260620 23:11:47.896023 19187 ts_tablet_manager.cc:1918] T d42afbe68b8b44d98234b7f3dd846c08 P 663f3caaf3a24449ae9075cd10afb8c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:11:47.896256 19187 log.cc:1196] T d42afbe68b8b44d98234b7f3dd846c08 P 663f3caaf3a24449ae9075cd10afb8c7: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-2/wal/wals/d42afbe68b8b44d98234b7f3dd846c08
I20260620 23:11:47.896795 19187 ts_tablet_manager.cc:1939] T d42afbe68b8b44d98234b7f3dd846c08 P 663f3caaf3a24449ae9075cd10afb8c7: Deleting consensus metadata
I20260620 23:11:47.897979 18590 catalog_manager.cc:4841] TS 663f3caaf3a24449ae9075cd10afb8c7 (127.10.191.195:42731): tablet d42afbe68b8b44d98234b7f3dd846c08 (table test-table [id=ba70a2efc6334555b3bc0ce5f8d87a31]) successfully deleted
I20260620 23:11:47.899410 19188 tablet_replica.cc:330] T d42afbe68b8b44d98234b7f3dd846c08 P e94d40b28f5d4842819973b4fb7b8272: stopping tablet replica
I20260620 23:11:47.899690 19188 raft_consensus.cc:2229] T d42afbe68b8b44d98234b7f3dd846c08 P e94d40b28f5d4842819973b4fb7b8272 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:11:47.900218 19189 tablet_replica.cc:330] T d42afbe68b8b44d98234b7f3dd846c08 P bfe128cc990a400b88234db04775deec: stopping tablet replica
I20260620 23:11:47.900794 19188 raft_consensus.cc:2258] T d42afbe68b8b44d98234b7f3dd846c08 P e94d40b28f5d4842819973b4fb7b8272 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:11:47.901096 19189 raft_consensus.cc:2229] T d42afbe68b8b44d98234b7f3dd846c08 P bfe128cc990a400b88234db04775deec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:11:47.901417 19189 raft_consensus.cc:2258] T d42afbe68b8b44d98234b7f3dd846c08 P bfe128cc990a400b88234db04775deec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:11:47.902535 19188 ts_tablet_manager.cc:1905] T d42afbe68b8b44d98234b7f3dd846c08 P e94d40b28f5d4842819973b4fb7b8272: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:11:47.904109 19189 ts_tablet_manager.cc:1905] T d42afbe68b8b44d98234b7f3dd846c08 P bfe128cc990a400b88234db04775deec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:11:47.911589 19188 ts_tablet_manager.cc:1918] T d42afbe68b8b44d98234b7f3dd846c08 P e94d40b28f5d4842819973b4fb7b8272: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:11:47.919330 19189 ts_tablet_manager.cc:1918] T d42afbe68b8b44d98234b7f3dd846c08 P bfe128cc990a400b88234db04775deec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:11:47.919646 19188 log.cc:1196] T d42afbe68b8b44d98234b7f3dd846c08 P e94d40b28f5d4842819973b4fb7b8272: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-0/wal/wals/d42afbe68b8b44d98234b7f3dd846c08
I20260620 23:11:47.920178 19188 ts_tablet_manager.cc:1939] T d42afbe68b8b44d98234b7f3dd846c08 P e94d40b28f5d4842819973b4fb7b8272: Deleting consensus metadata
I20260620 23:11:47.920339 19189 log.cc:1196] T d42afbe68b8b44d98234b7f3dd846c08 P bfe128cc990a400b88234db04775deec: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-1/wal/wals/d42afbe68b8b44d98234b7f3dd846c08
I20260620 23:11:47.920861 19189 ts_tablet_manager.cc:1939] T d42afbe68b8b44d98234b7f3dd846c08 P bfe128cc990a400b88234db04775deec: Deleting consensus metadata
I20260620 23:11:47.922358 18590 catalog_manager.cc:4841] TS e94d40b28f5d4842819973b4fb7b8272 (127.10.191.193:37597): tablet d42afbe68b8b44d98234b7f3dd846c08 (table test-table [id=ba70a2efc6334555b3bc0ce5f8d87a31]) successfully deleted
I20260620 23:11:47.923661 18593 catalog_manager.cc:4841] TS bfe128cc990a400b88234db04775deec (127.10.191.194:37179): tablet d42afbe68b8b44d98234b7f3dd846c08 (table test-table [id=ba70a2efc6334555b3bc0ce5f8d87a31]) successfully deleted
Jun 20 23:11:47 dist-test-slave-jkbq krb5kdc[18561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 23:11:47 dist-test-slave-jkbq krb5kdc[18561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:11:47 dist-test-slave-jkbq krb5kdc[18561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 23:11:47.962546 19196 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:11:47.948637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58010 (local address 127.10.191.254:36897)
0620 23:11:47.949139 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:47.949146 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:47.949749 (+ 603us) server_negotiation.cc:407] Connection header received
0620 23:11:47.949951 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:47.949956 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:47.950037 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:47.950154 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:47.951917 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.953119 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.955119 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.955826 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.955932 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:47.962223 (+ 6291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58010: BlockingRecv error: recv got EOF from 127.0.0.1:58010 (error 108)
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":240,"threads_started":1}
Jun 20 23:11:47 dist-test-slave-jkbq krb5kdc[18561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM
I20260620 23:11:48.010780 19196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:47.992443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58014 (local address 127.10.191.254:36897)
0620 23:11:47.992621 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:47.992627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:47.993148 (+ 521us) server_negotiation.cc:407] Connection header received
0620 23:11:47.993346 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:47.993353 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:47.993437 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:47.993552 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:47.995200 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.996396 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.998205 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:47.998900 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:47.999025 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:48.004038 (+ 5013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:48.004045 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:48.004058 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:48.004102 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:48.007776 (+ 3674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:48.008639 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:48.008645 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:48.008649 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:48.008805 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:48.009253 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:48.009259 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:48.009262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:48.009605 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:48.009859 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:48.010197 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 23:11:48.010419 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 23:11:48.015002 18606 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58014:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:11:48.015718 18606 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:11:48.023494 18743 tablet_service.cc:1467] Processing CreateTablet for tablet 27f98a55c0a74adb8bd46dd632e5c351 (DEFAULT_TABLE table=test-table [id=e817996cd78943209e534549411cfde4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:11:48.023943 18743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27f98a55c0a74adb8bd46dd632e5c351. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:11:48.024009 18882 tablet_service.cc:1467] Processing CreateTablet for tablet 27f98a55c0a74adb8bd46dd632e5c351 (DEFAULT_TABLE table=test-table [id=e817996cd78943209e534549411cfde4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:11:48.024454 18882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27f98a55c0a74adb8bd46dd632e5c351. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:11:48.027294 19021 tablet_service.cc:1467] Processing CreateTablet for tablet 27f98a55c0a74adb8bd46dd632e5c351 (DEFAULT_TABLE table=test-table [id=e817996cd78943209e534549411cfde4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:11:48.027690 19021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27f98a55c0a74adb8bd46dd632e5c351. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:11:48.034301 19203 tablet_bootstrap.cc:492] T 27f98a55c0a74adb8bd46dd632e5c351 P e94d40b28f5d4842819973b4fb7b8272: Bootstrap starting.
I20260620 23:11:48.036504 19203 tablet_bootstrap.cc:654] T 27f98a55c0a74adb8bd46dd632e5c351 P e94d40b28f5d4842819973b4fb7b8272: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:48.053850 19206 tablet_bootstrap.cc:492] T 27f98a55c0a74adb8bd46dd632e5c351 P bfe128cc990a400b88234db04775deec: Bootstrap starting.
I20260620 23:11:48.055195 19205 tablet_bootstrap.cc:492] T 27f98a55c0a74adb8bd46dd632e5c351 P 663f3caaf3a24449ae9075cd10afb8c7: Bootstrap starting.
I20260620 23:11:48.058550 19206 tablet_bootstrap.cc:654] T 27f98a55c0a74adb8bd46dd632e5c351 P bfe128cc990a400b88234db04775deec: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:48.062173 19205 tablet_bootstrap.cc:654] T 27f98a55c0a74adb8bd46dd632e5c351 P 663f3caaf3a24449ae9075cd10afb8c7: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:48.064913 19206 tablet_bootstrap.cc:492] T 27f98a55c0a74adb8bd46dd632e5c351 P bfe128cc990a400b88234db04775deec: No bootstrap required, opened a new log
I20260620 23:11:48.065085 19206 ts_tablet_manager.cc:1397] T 27f98a55c0a74adb8bd46dd632e5c351 P bfe128cc990a400b88234db04775deec: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.003s
I20260620 23:11:48.065989 19206 raft_consensus.cc:348] T 27f98a55c0a74adb8bd46dd632e5c351 P bfe128cc990a400b88234db04775deec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } }
I20260620 23:11:48.066264 19206 raft_consensus.cc:374] T 27f98a55c0a74adb8bd46dd632e5c351 P bfe128cc990a400b88234db04775deec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:48.066361 19206 raft_consensus.cc:729] T 27f98a55c0a74adb8bd46dd632e5c351 P bfe128cc990a400b88234db04775deec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfe128cc990a400b88234db04775deec, State: Initialized, Role: FOLLOWER
I20260620 23:11:48.067076 19206 consensus_queue.cc:260] T 27f98a55c0a74adb8bd46dd632e5c351 P bfe128cc990a400b88234db04775deec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } }
I20260620 23:11:48.068882 19206 ts_tablet_manager.cc:1428] T 27f98a55c0a74adb8bd46dd632e5c351 P bfe128cc990a400b88234db04775deec: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260620 23:11:48.069321 19205 tablet_bootstrap.cc:492] T 27f98a55c0a74adb8bd46dd632e5c351 P 663f3caaf3a24449ae9075cd10afb8c7: No bootstrap required, opened a new log
I20260620 23:11:48.069525 19205 ts_tablet_manager.cc:1397] T 27f98a55c0a74adb8bd46dd632e5c351 P 663f3caaf3a24449ae9075cd10afb8c7: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.002s
I20260620 23:11:48.069689 19203 tablet_bootstrap.cc:492] T 27f98a55c0a74adb8bd46dd632e5c351 P e94d40b28f5d4842819973b4fb7b8272: No bootstrap required, opened a new log
I20260620 23:11:48.069850 19203 ts_tablet_manager.cc:1397] T 27f98a55c0a74adb8bd46dd632e5c351 P e94d40b28f5d4842819973b4fb7b8272: Time spent bootstrapping tablet: real 0.036s user 0.008s sys 0.024s
I20260620 23:11:48.070467 19205 raft_consensus.cc:348] T 27f98a55c0a74adb8bd46dd632e5c351 P 663f3caaf3a24449ae9075cd10afb8c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } }
I20260620 23:11:48.070770 19205 raft_consensus.cc:374] T 27f98a55c0a74adb8bd46dd632e5c351 P 663f3caaf3a24449ae9075cd10afb8c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:48.070655 19203 raft_consensus.cc:348] T 27f98a55c0a74adb8bd46dd632e5c351 P e94d40b28f5d4842819973b4fb7b8272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } }
I20260620 23:11:48.071038 19203 raft_consensus.cc:374] T 27f98a55c0a74adb8bd46dd632e5c351 P e94d40b28f5d4842819973b4fb7b8272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:48.071170 19203 raft_consensus.cc:729] T 27f98a55c0a74adb8bd46dd632e5c351 P e94d40b28f5d4842819973b4fb7b8272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e94d40b28f5d4842819973b4fb7b8272, State: Initialized, Role: FOLLOWER
I20260620 23:11:48.070973 19205 raft_consensus.cc:729] T 27f98a55c0a74adb8bd46dd632e5c351 P 663f3caaf3a24449ae9075cd10afb8c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663f3caaf3a24449ae9075cd10afb8c7, State: Initialized, Role: FOLLOWER
I20260620 23:11:48.071419 19203 consensus_queue.cc:260] T 27f98a55c0a74adb8bd46dd632e5c351 P e94d40b28f5d4842819973b4fb7b8272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } }
I20260620 23:11:48.072009 19203 ts_tablet_manager.cc:1428] T 27f98a55c0a74adb8bd46dd632e5c351 P e94d40b28f5d4842819973b4fb7b8272: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:11:48.072302 19205 consensus_queue.cc:260] T 27f98a55c0a74adb8bd46dd632e5c351 P 663f3caaf3a24449ae9075cd10afb8c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } }
I20260620 23:11:48.073443 19205 ts_tablet_manager.cc:1428] T 27f98a55c0a74adb8bd46dd632e5c351 P 663f3caaf3a24449ae9075cd10afb8c7: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260620 23:11:48.363428 19122 raft_consensus.cc:482] T 27f98a55c0a74adb8bd46dd632e5c351 P 663f3caaf3a24449ae9075cd10afb8c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:11:48.363600 19122 raft_consensus.cc:504] T 27f98a55c0a74adb8bd46dd632e5c351 P 663f3caaf3a24449ae9075cd10afb8c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } }
I20260620 23:11:48.364300 19122 leader_election.cc:288] T 27f98a55c0a74adb8bd46dd632e5c351 P 663f3caaf3a24449ae9075cd10afb8c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e94d40b28f5d4842819973b4fb7b8272 (127.10.191.193:37597), bfe128cc990a400b88234db04775deec (127.10.191.194:37179)
I20260620 23:11:48.364861 18901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27f98a55c0a74adb8bd46dd632e5c351" candidate_uuid: "663f3caaf3a24449ae9075cd10afb8c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfe128cc990a400b88234db04775deec" is_pre_election: true
I20260620 23:11:48.364899 18768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27f98a55c0a74adb8bd46dd632e5c351" candidate_uuid: "663f3caaf3a24449ae9075cd10afb8c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e94d40b28f5d4842819973b4fb7b8272" is_pre_election: true
I20260620 23:11:48.365118 18901 raft_consensus.cc:2454] T 27f98a55c0a74adb8bd46dd632e5c351 P bfe128cc990a400b88234db04775deec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 663f3caaf3a24449ae9075cd10afb8c7 in term 0.
I20260620 23:11:48.365118 18768 raft_consensus.cc:2454] T 27f98a55c0a74adb8bd46dd632e5c351 P e94d40b28f5d4842819973b4fb7b8272 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 663f3caaf3a24449ae9075cd10afb8c7 in term 0.
I20260620 23:11:48.365586 18972 leader_election.cc:302] T 27f98a55c0a74adb8bd46dd632e5c351 P 663f3caaf3a24449ae9075cd10afb8c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 663f3caaf3a24449ae9075cd10afb8c7, bfe128cc990a400b88234db04775deec, e94d40b28f5d4842819973b4fb7b8272; no voters:
I20260620 23:11:48.365837 19122 raft_consensus.cc:2781] T 27f98a55c0a74adb8bd46dd632e5c351 P 663f3caaf3a24449ae9075cd10afb8c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:11:48.365957 19122 raft_consensus.cc:482] T 27f98a55c0a74adb8bd46dd632e5c351 P 663f3caaf3a24449ae9075cd10afb8c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:11:48.366058 19122 raft_consensus.cc:3037] T 27f98a55c0a74adb8bd46dd632e5c351 P 663f3caaf3a24449ae9075cd10afb8c7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:48.367983 19122 raft_consensus.cc:504] T 27f98a55c0a74adb8bd46dd632e5c351 P 663f3caaf3a24449ae9075cd10afb8c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } }
I20260620 23:11:48.368506 19122 leader_election.cc:288] T 27f98a55c0a74adb8bd46dd632e5c351 P 663f3caaf3a24449ae9075cd10afb8c7 [CANDIDATE]: Term 1 election: Requested vote from peers e94d40b28f5d4842819973b4fb7b8272 (127.10.191.193:37597), bfe128cc990a400b88234db04775deec (127.10.191.194:37179)
I20260620 23:11:48.368939 18768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27f98a55c0a74adb8bd46dd632e5c351" candidate_uuid: "663f3caaf3a24449ae9075cd10afb8c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e94d40b28f5d4842819973b4fb7b8272"
I20260620 23:11:48.368983 18901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27f98a55c0a74adb8bd46dd632e5c351" candidate_uuid: "663f3caaf3a24449ae9075cd10afb8c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfe128cc990a400b88234db04775deec"
I20260620 23:11:48.369115 18768 raft_consensus.cc:3037] T 27f98a55c0a74adb8bd46dd632e5c351 P e94d40b28f5d4842819973b4fb7b8272 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:48.369230 18901 raft_consensus.cc:3037] T 27f98a55c0a74adb8bd46dd632e5c351 P bfe128cc990a400b88234db04775deec [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:48.371601 18768 raft_consensus.cc:2454] T 27f98a55c0a74adb8bd46dd632e5c351 P e94d40b28f5d4842819973b4fb7b8272 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663f3caaf3a24449ae9075cd10afb8c7 in term 1.
I20260620 23:11:48.371790 18901 raft_consensus.cc:2454] T 27f98a55c0a74adb8bd46dd632e5c351 P bfe128cc990a400b88234db04775deec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663f3caaf3a24449ae9075cd10afb8c7 in term 1.
I20260620 23:11:48.372002 18972 leader_election.cc:302] T 27f98a55c0a74adb8bd46dd632e5c351 P 663f3caaf3a24449ae9075cd10afb8c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 663f3caaf3a24449ae9075cd10afb8c7, e94d40b28f5d4842819973b4fb7b8272; no voters:
I20260620 23:11:48.372272 19122 raft_consensus.cc:2781] T 27f98a55c0a74adb8bd46dd632e5c351 P 663f3caaf3a24449ae9075cd10afb8c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:11:48.372617 19122 raft_consensus.cc:686] T 27f98a55c0a74adb8bd46dd632e5c351 P 663f3caaf3a24449ae9075cd10afb8c7 [term 1 LEADER]: Becoming Leader. State: Replica: 663f3caaf3a24449ae9075cd10afb8c7, State: Running, Role: LEADER
I20260620 23:11:48.372905 19122 consensus_queue.cc:237] T 27f98a55c0a74adb8bd46dd632e5c351 P 663f3caaf3a24449ae9075cd10afb8c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } }
I20260620 23:11:48.375383 18606 catalog_manager.cc:5495] T 27f98a55c0a74adb8bd46dd632e5c351 P 663f3caaf3a24449ae9075cd10afb8c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 663f3caaf3a24449ae9075cd10afb8c7 (127.10.191.195). New cstate: current_term: 1 leader_uuid: "663f3caaf3a24449ae9075cd10afb8c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } health_report { overall_health: HEALTHY } } }
I20260620 23:11:48.419450 19212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:48.414657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52012 (local address 127.10.191.195:42731)
0620 23:11:48.415214 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:48.415218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:48.415234 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:11:48.415289 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:48.415292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:48.415351 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:48.415433 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:11:48.415729 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:48.416437 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:48.417753 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:48.418157 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:48.418257 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:48.418667 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:11:48.418782 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:11:48.418850 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:48.419031 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 23:11:48.419093 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":370,"threads_started":1}
I20260620 23:11:48.422173 18768 raft_consensus.cc:1261] T 27f98a55c0a74adb8bd46dd632e5c351 P e94d40b28f5d4842819973b4fb7b8272 [term 1 FOLLOWER]: Refusing update from remote peer 663f3caaf3a24449ae9075cd10afb8c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:11:48.422210 18901 raft_consensus.cc:1261] T 27f98a55c0a74adb8bd46dd632e5c351 P bfe128cc990a400b88234db04775deec [term 1 FOLLOWER]: Refusing update from remote peer 663f3caaf3a24449ae9075cd10afb8c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:11:48.422812 19209 consensus_queue.cc:1035] T 27f98a55c0a74adb8bd46dd632e5c351 P 663f3caaf3a24449ae9075cd10afb8c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:11:48.423216 19122 consensus_queue.cc:1035] T 27f98a55c0a74adb8bd46dd632e5c351 P 663f3caaf3a24449ae9075cd10afb8c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:11:48.441730 18606 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:11:48.441972 18606 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:11:48.444573 18606 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 61bc181e1aab415b945d91121993c454: Sending DeleteTablet for 3 replicas of tablet 27f98a55c0a74adb8bd46dd632e5c351
I20260620 23:11:48.445267 18743 tablet_service.cc:1514] Processing DeleteTablet for tablet 27f98a55c0a74adb8bd46dd632e5c351 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:11:48 UTC) from {username='oryx', principal='oryx/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:41416
I20260620 23:11:48.445360 19021 tablet_service.cc:1514] Processing DeleteTablet for tablet 27f98a55c0a74adb8bd46dd632e5c351 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:11:48 UTC) from {username='oryx', principal='oryx/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:51996
I20260620 23:11:48.445346 18882 tablet_service.cc:1514] Processing DeleteTablet for tablet 27f98a55c0a74adb8bd46dd632e5c351 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:11:48 UTC) from {username='oryx', principal='oryx/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:55916
I20260620 23:11:48.445909 19222 tablet_replica.cc:330] T 27f98a55c0a74adb8bd46dd632e5c351 P e94d40b28f5d4842819973b4fb7b8272: stopping tablet replica
I20260620 23:11:48.446214 19222 raft_consensus.cc:2229] T 27f98a55c0a74adb8bd46dd632e5c351 P e94d40b28f5d4842819973b4fb7b8272 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:11:48.446666 19223 tablet_replica.cc:330] T 27f98a55c0a74adb8bd46dd632e5c351 P 663f3caaf3a24449ae9075cd10afb8c7: stopping tablet replica
I20260620 23:11:48.446853 19223 raft_consensus.cc:2229] T 27f98a55c0a74adb8bd46dd632e5c351 P 663f3caaf3a24449ae9075cd10afb8c7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:11:48.447031 19222 raft_consensus.cc:2258] T 27f98a55c0a74adb8bd46dd632e5c351 P e94d40b28f5d4842819973b4fb7b8272 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:11:48.446993 18606 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58014:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 23:11:48.447274 19223 raft_consensus.cc:2258] T 27f98a55c0a74adb8bd46dd632e5c351 P 663f3caaf3a24449ae9075cd10afb8c7 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 23:11:48.447621 18606 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:11:48.448421 19223 ts_tablet_manager.cc:1905] T 27f98a55c0a74adb8bd46dd632e5c351 P 663f3caaf3a24449ae9075cd10afb8c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:11:48.448655 19222 ts_tablet_manager.cc:1905] T 27f98a55c0a74adb8bd46dd632e5c351 P e94d40b28f5d4842819973b4fb7b8272: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:11:48.454823 19223 ts_tablet_manager.cc:1918] T 27f98a55c0a74adb8bd46dd632e5c351 P 663f3caaf3a24449ae9075cd10afb8c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:11:48.454977 19223 log.cc:1196] T 27f98a55c0a74adb8bd46dd632e5c351 P 663f3caaf3a24449ae9075cd10afb8c7: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-2/wal/wals/27f98a55c0a74adb8bd46dd632e5c351
I20260620 23:11:48.455441 19223 ts_tablet_manager.cc:1939] T 27f98a55c0a74adb8bd46dd632e5c351 P 663f3caaf3a24449ae9075cd10afb8c7: Deleting consensus metadata
I20260620 23:11:48.455874 18881 tablet_service.cc:1467] Processing CreateTablet for tablet 3e2fa0b0ff444a7fb2e2242d9ce437dd (DEFAULT_TABLE table=test-table [id=20fdd787278f413789c46eecb47db89b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:11:48.456285 18881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e2fa0b0ff444a7fb2e2242d9ce437dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:11:48.457918 19222 ts_tablet_manager.cc:1918] T 27f98a55c0a74adb8bd46dd632e5c351 P e94d40b28f5d4842819973b4fb7b8272: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:11:48.458062 19222 log.cc:1196] T 27f98a55c0a74adb8bd46dd632e5c351 P e94d40b28f5d4842819973b4fb7b8272: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-0/wal/wals/27f98a55c0a74adb8bd46dd632e5c351
I20260620 23:11:48.458581 19222 ts_tablet_manager.cc:1939] T 27f98a55c0a74adb8bd46dd632e5c351 P e94d40b28f5d4842819973b4fb7b8272: Deleting consensus metadata
I20260620 23:11:48.459326 18590 catalog_manager.cc:4841] TS 663f3caaf3a24449ae9075cd10afb8c7 (127.10.191.195:42731): tablet 27f98a55c0a74adb8bd46dd632e5c351 (table test-table [id=e817996cd78943209e534549411cfde4]) successfully deleted
I20260620 23:11:48.460135 18743 tablet_service.cc:1467] Processing CreateTablet for tablet 3e2fa0b0ff444a7fb2e2242d9ce437dd (DEFAULT_TABLE table=test-table [id=20fdd787278f413789c46eecb47db89b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:11:48.460223 19021 tablet_service.cc:1467] Processing CreateTablet for tablet 3e2fa0b0ff444a7fb2e2242d9ce437dd (DEFAULT_TABLE table=test-table [id=20fdd787278f413789c46eecb47db89b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:11:48.460523 18743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e2fa0b0ff444a7fb2e2242d9ce437dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:11:48.460680 19021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e2fa0b0ff444a7fb2e2242d9ce437dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:11:48.460901 19224 tablet_replica.cc:330] T 27f98a55c0a74adb8bd46dd632e5c351 P bfe128cc990a400b88234db04775deec: stopping tablet replica
I20260620 23:11:48.461105 19224 raft_consensus.cc:2229] T 27f98a55c0a74adb8bd46dd632e5c351 P bfe128cc990a400b88234db04775deec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:11:48.461356 19224 raft_consensus.cc:2258] T 27f98a55c0a74adb8bd46dd632e5c351 P bfe128cc990a400b88234db04775deec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:11:48.462483 19206 tablet_bootstrap.cc:492] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P bfe128cc990a400b88234db04775deec: Bootstrap starting.
I20260620 23:11:48.463195 18590 catalog_manager.cc:4841] TS e94d40b28f5d4842819973b4fb7b8272 (127.10.191.193:37597): tablet 27f98a55c0a74adb8bd46dd632e5c351 (table test-table [id=e817996cd78943209e534549411cfde4]) successfully deleted
I20260620 23:11:48.464147 19206 tablet_bootstrap.cc:654] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P bfe128cc990a400b88234db04775deec: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:48.464534 19224 ts_tablet_manager.cc:1905] T 27f98a55c0a74adb8bd46dd632e5c351 P bfe128cc990a400b88234db04775deec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:11:48.467046 19205 tablet_bootstrap.cc:492] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P 663f3caaf3a24449ae9075cd10afb8c7: Bootstrap starting.
I20260620 23:11:48.469120 19205 tablet_bootstrap.cc:654] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P 663f3caaf3a24449ae9075cd10afb8c7: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:48.469215 19203 tablet_bootstrap.cc:492] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P e94d40b28f5d4842819973b4fb7b8272: Bootstrap starting.
I20260620 23:11:48.470759 19203 tablet_bootstrap.cc:654] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P e94d40b28f5d4842819973b4fb7b8272: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:48.471220 19224 ts_tablet_manager.cc:1918] T 27f98a55c0a74adb8bd46dd632e5c351 P bfe128cc990a400b88234db04775deec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:11:48.471402 19224 log.cc:1196] T 27f98a55c0a74adb8bd46dd632e5c351 P bfe128cc990a400b88234db04775deec: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-1/wal/wals/27f98a55c0a74adb8bd46dd632e5c351
I20260620 23:11:48.471901 19224 ts_tablet_manager.cc:1939] T 27f98a55c0a74adb8bd46dd632e5c351 P bfe128cc990a400b88234db04775deec: Deleting consensus metadata
I20260620 23:11:48.473026 18593 catalog_manager.cc:4841] TS bfe128cc990a400b88234db04775deec (127.10.191.194:37179): tablet 27f98a55c0a74adb8bd46dd632e5c351 (table test-table [id=e817996cd78943209e534549411cfde4]) successfully deleted
I20260620 23:11:48.474913 19203 tablet_bootstrap.cc:492] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P e94d40b28f5d4842819973b4fb7b8272: No bootstrap required, opened a new log
I20260620 23:11:48.475085 19203 ts_tablet_manager.cc:1397] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P e94d40b28f5d4842819973b4fb7b8272: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 23:11:48.475469 19205 tablet_bootstrap.cc:492] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P 663f3caaf3a24449ae9075cd10afb8c7: No bootstrap required, opened a new log
I20260620 23:11:48.475636 19205 ts_tablet_manager.cc:1397] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P 663f3caaf3a24449ae9075cd10afb8c7: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20260620 23:11:48.475883 19206 tablet_bootstrap.cc:492] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P bfe128cc990a400b88234db04775deec: No bootstrap required, opened a new log
I20260620 23:11:48.476012 19206 ts_tablet_manager.cc:1397] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P bfe128cc990a400b88234db04775deec: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.003s
I20260620 23:11:48.476442 19205 raft_consensus.cc:348] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P 663f3caaf3a24449ae9075cd10afb8c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } }
I20260620 23:11:48.476420 19203 raft_consensus.cc:348] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P e94d40b28f5d4842819973b4fb7b8272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } }
I20260620 23:11:48.476678 19205 raft_consensus.cc:374] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P 663f3caaf3a24449ae9075cd10afb8c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:48.476699 19203 raft_consensus.cc:374] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P e94d40b28f5d4842819973b4fb7b8272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:48.476779 19203 raft_consensus.cc:729] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P e94d40b28f5d4842819973b4fb7b8272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e94d40b28f5d4842819973b4fb7b8272, State: Initialized, Role: FOLLOWER
I20260620 23:11:48.476775 19205 raft_consensus.cc:729] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P 663f3caaf3a24449ae9075cd10afb8c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663f3caaf3a24449ae9075cd10afb8c7, State: Initialized, Role: FOLLOWER
I20260620 23:11:48.476774 19206 raft_consensus.cc:348] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P bfe128cc990a400b88234db04775deec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } }
I20260620 23:11:48.476996 19206 raft_consensus.cc:374] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P bfe128cc990a400b88234db04775deec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:48.477082 19206 raft_consensus.cc:729] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P bfe128cc990a400b88234db04775deec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfe128cc990a400b88234db04775deec, State: Initialized, Role: FOLLOWER
I20260620 23:11:48.477010 19205 consensus_queue.cc:260] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P 663f3caaf3a24449ae9075cd10afb8c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } }
I20260620 23:11:48.477036 19203 consensus_queue.cc:260] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P e94d40b28f5d4842819973b4fb7b8272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } }
I20260620 23:11:48.477326 19206 consensus_queue.cc:260] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P bfe128cc990a400b88234db04775deec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } }
I20260620 23:11:48.477556 19205 ts_tablet_manager.cc:1428] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P 663f3caaf3a24449ae9075cd10afb8c7: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260620 23:11:48.478732 19206 ts_tablet_manager.cc:1428] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P bfe128cc990a400b88234db04775deec: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 23:11:48.479518 19203 ts_tablet_manager.cc:1428] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P e94d40b28f5d4842819973b4fb7b8272: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260620 23:11:48.733392 19121 raft_consensus.cc:482] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P bfe128cc990a400b88234db04775deec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:11:48.733685 19121 raft_consensus.cc:504] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P bfe128cc990a400b88234db04775deec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } }
I20260620 23:11:48.734333 19121 leader_election.cc:288] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P bfe128cc990a400b88234db04775deec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e94d40b28f5d4842819973b4fb7b8272 (127.10.191.193:37597), 663f3caaf3a24449ae9075cd10afb8c7 (127.10.191.195:42731)
I20260620 23:11:48.735083 18768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e2fa0b0ff444a7fb2e2242d9ce437dd" candidate_uuid: "bfe128cc990a400b88234db04775deec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e94d40b28f5d4842819973b4fb7b8272" is_pre_election: true
I20260620 23:11:48.735113 19043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e2fa0b0ff444a7fb2e2242d9ce437dd" candidate_uuid: "bfe128cc990a400b88234db04775deec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663f3caaf3a24449ae9075cd10afb8c7" is_pre_election: true
I20260620 23:11:48.735329 18768 raft_consensus.cc:2454] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P e94d40b28f5d4842819973b4fb7b8272 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfe128cc990a400b88234db04775deec in term 0.
I20260620 23:11:48.735426 19043 raft_consensus.cc:2454] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P 663f3caaf3a24449ae9075cd10afb8c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfe128cc990a400b88234db04775deec in term 0.
I20260620 23:11:48.735817 18833 leader_election.cc:302] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P bfe128cc990a400b88234db04775deec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bfe128cc990a400b88234db04775deec, e94d40b28f5d4842819973b4fb7b8272; no voters:
I20260620 23:11:48.736090 19121 raft_consensus.cc:2781] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P bfe128cc990a400b88234db04775deec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:11:48.736173 19121 raft_consensus.cc:482] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P bfe128cc990a400b88234db04775deec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:11:48.736297 19121 raft_consensus.cc:3037] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P bfe128cc990a400b88234db04775deec [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:48.738240 19121 raft_consensus.cc:504] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P bfe128cc990a400b88234db04775deec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } }
I20260620 23:11:48.738758 19121 leader_election.cc:288] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P bfe128cc990a400b88234db04775deec [CANDIDATE]: Term 1 election: Requested vote from peers e94d40b28f5d4842819973b4fb7b8272 (127.10.191.193:37597), 663f3caaf3a24449ae9075cd10afb8c7 (127.10.191.195:42731)
I20260620 23:11:48.739231 18768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e2fa0b0ff444a7fb2e2242d9ce437dd" candidate_uuid: "bfe128cc990a400b88234db04775deec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e94d40b28f5d4842819973b4fb7b8272"
I20260620 23:11:48.739260 19043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e2fa0b0ff444a7fb2e2242d9ce437dd" candidate_uuid: "bfe128cc990a400b88234db04775deec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663f3caaf3a24449ae9075cd10afb8c7"
I20260620 23:11:48.739395 18768 raft_consensus.cc:3037] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P e94d40b28f5d4842819973b4fb7b8272 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:48.739449 19043 raft_consensus.cc:3037] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P 663f3caaf3a24449ae9075cd10afb8c7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:48.741315 19043 raft_consensus.cc:2454] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P 663f3caaf3a24449ae9075cd10afb8c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfe128cc990a400b88234db04775deec in term 1.
I20260620 23:11:48.741333 18768 raft_consensus.cc:2454] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P e94d40b28f5d4842819973b4fb7b8272 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfe128cc990a400b88234db04775deec in term 1.
I20260620 23:11:48.741696 18833 leader_election.cc:302] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P bfe128cc990a400b88234db04775deec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bfe128cc990a400b88234db04775deec, e94d40b28f5d4842819973b4fb7b8272; no voters:
I20260620 23:11:48.741918 19121 raft_consensus.cc:2781] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P bfe128cc990a400b88234db04775deec [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:11:48.742295 19121 raft_consensus.cc:686] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P bfe128cc990a400b88234db04775deec [term 1 LEADER]: Becoming Leader. State: Replica: bfe128cc990a400b88234db04775deec, State: Running, Role: LEADER
I20260620 23:11:48.742579 19121 consensus_queue.cc:237] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P bfe128cc990a400b88234db04775deec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } }
I20260620 23:11:48.744926 18604 catalog_manager.cc:5495] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P bfe128cc990a400b88234db04775deec reported cstate change: term changed from 0 to 1, leader changed from <none> to bfe128cc990a400b88234db04775deec (127.10.191.194). New cstate: current_term: 1 leader_uuid: "bfe128cc990a400b88234db04775deec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfe128cc990a400b88234db04775deec" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 } health_report { overall_health: UNKNOWN } } }
I20260620 23:11:48.776328 19231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:48.771235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55928 (local address 127.10.191.194:37179)
0620 23:11:48.771768 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:48.771773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:48.771799 (+ 26us) server_negotiation.cc:407] Connection header received
0620 23:11:48.771843 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:48.771847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:48.771905 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:48.772010 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:11:48.772430 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:48.773160 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:48.774463 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:48.774943 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:48.775086 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:48.775540 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:11:48.775639 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:11:48.775740 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:48.775899 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 23:11:48.775970 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":269,"threads_started":1}
I20260620 23:11:48.788712 19231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:48.783871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:44407 (local address 127.10.191.194:37179)
0620 23:11:48.784037 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:48.784040 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:48.784433 (+ 393us) server_negotiation.cc:407] Connection header received
0620 23:11:48.784561 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:48.784565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:48.784628 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:48.784703 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:11:48.785057 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:48.785731 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:48.786956 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:48.787358 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:48.787541 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:48.787944 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:11:48.788033 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:11:48.788128 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:48.788252 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 23:11:48.788368 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":66}
I20260620 23:11:48.788803 19234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:48.783790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.194:37179 (local address 127.10.191.194:44407)
0620 23:11:48.784346 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:48.784365 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:48.784462 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:48.784735 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:48.784738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:48.784747 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:11:48.784944 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:48.784951 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:48.785829 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:48.785832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:48.786767 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:48.786777 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:48.787557 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:48.787563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:48.787762 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:48.787794 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:11:48.788126 (+ 332us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:11:48.788136 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 23:11:48.788236 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 23:11:48.788406 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"mutex_wait_us":76,"thread_start_us":313,"threads_started":1}
I20260620 23:11:48.795768 19043 raft_consensus.cc:1261] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P 663f3caaf3a24449ae9075cd10afb8c7 [term 1 FOLLOWER]: Refusing update from remote peer bfe128cc990a400b88234db04775deec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:11:48.795929 18768 raft_consensus.cc:1261] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P e94d40b28f5d4842819973b4fb7b8272 [term 1 FOLLOWER]: Refusing update from remote peer bfe128cc990a400b88234db04775deec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:11:48.796361 19228 consensus_queue.cc:1035] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P bfe128cc990a400b88234db04775deec [LEADER]: Connected to new peer: Peer: permanent_uuid: "663f3caaf3a24449ae9075cd10afb8c7" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 42731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:11:48.796684 19121 consensus_queue.cc:1035] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P bfe128cc990a400b88234db04775deec [LEADER]: Connected to new peer: Peer: permanent_uuid: "e94d40b28f5d4842819973b4fb7b8272" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 37597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:11:48.843007 18605 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:11:48.843321 18605 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:11:48.846459 18605 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 61bc181e1aab415b945d91121993c454: Sending DeleteTablet for 3 replicas of tablet 3e2fa0b0ff444a7fb2e2242d9ce437dd
I20260620 23:11:48.847267 18743 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e2fa0b0ff444a7fb2e2242d9ce437dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:11:48 UTC) from {username='oryx', principal='oryx/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:41416
I20260620 23:11:48.847638 19222 tablet_replica.cc:330] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P e94d40b28f5d4842819973b4fb7b8272: stopping tablet replica
I20260620 23:11:48.847712 18881 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e2fa0b0ff444a7fb2e2242d9ce437dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:11:48 UTC) from {username='oryx', principal='oryx/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:55916
I20260620 23:11:48.847709 19021 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e2fa0b0ff444a7fb2e2242d9ce437dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:11:48 UTC) from {username='oryx', principal='oryx/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:51996
I20260620 23:11:48.847851 19222 raft_consensus.cc:2229] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P e94d40b28f5d4842819973b4fb7b8272 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:11:48.848030 19224 tablet_replica.cc:330] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P bfe128cc990a400b88234db04775deec: stopping tablet replica
I20260620 23:11:48.848080 19222 raft_consensus.cc:2258] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P e94d40b28f5d4842819973b4fb7b8272 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:11:48.848261 19224 raft_consensus.cc:2229] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P bfe128cc990a400b88234db04775deec [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:11:48.848771 19224 raft_consensus.cc:2258] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P bfe128cc990a400b88234db04775deec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:11:48.848865 19223 tablet_replica.cc:330] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P 663f3caaf3a24449ae9075cd10afb8c7: stopping tablet replica
I20260620 23:11:48.849042 19223 raft_consensus.cc:2229] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P 663f3caaf3a24449ae9075cd10afb8c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:11:48.849211 19222 ts_tablet_manager.cc:1905] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P e94d40b28f5d4842819973b4fb7b8272: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:11:48.849228 19223 raft_consensus.cc:2258] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P 663f3caaf3a24449ae9075cd10afb8c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:11:48.849895 19224 ts_tablet_manager.cc:1905] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P bfe128cc990a400b88234db04775deec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:11:48.850765 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 18677
I20260620 23:11:48.856842 19224 ts_tablet_manager.cc:1918] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P bfe128cc990a400b88234db04775deec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:11:48.857007 19224 log.cc:1196] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P bfe128cc990a400b88234db04775deec: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-1/wal/wals/3e2fa0b0ff444a7fb2e2242d9ce437dd
I20260620 23:11:48.857429 19224 ts_tablet_manager.cc:1939] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P bfe128cc990a400b88234db04775deec: Deleting consensus metadata
I20260620 23:11:48.858485 18593 catalog_manager.cc:4841] TS bfe128cc990a400b88234db04775deec (127.10.191.194:37179): tablet 3e2fa0b0ff444a7fb2e2242d9ce437dd (table test-table [id=20fdd787278f413789c46eecb47db89b]) successfully deleted
I20260620 23:11:48.859301 19222 ts_tablet_manager.cc:1918] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P e94d40b28f5d4842819973b4fb7b8272: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:11:48.859541 19222 log.cc:1196] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P e94d40b28f5d4842819973b4fb7b8272: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-0/wal/wals/3e2fa0b0ff444a7fb2e2242d9ce437dd
I20260620 23:11:48.860011 19222 ts_tablet_manager.cc:1939] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P e94d40b28f5d4842819973b4fb7b8272: Deleting consensus metadata
I20260620 23:11:48.860076 19223 ts_tablet_manager.cc:1905] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P 663f3caaf3a24449ae9075cd10afb8c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:11:48.863300 18590 catalog_manager.cc:4841] TS e94d40b28f5d4842819973b4fb7b8272 (127.10.191.193:37597): tablet 3e2fa0b0ff444a7fb2e2242d9ce437dd (table test-table [id=20fdd787278f413789c46eecb47db89b]) successfully deleted
I20260620 23:11:48.865216 19223 ts_tablet_manager.cc:1918] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P 663f3caaf3a24449ae9075cd10afb8c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:11:48.865381 19223 log.cc:1196] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P 663f3caaf3a24449ae9075cd10afb8c7: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781997042893763-11007-0/minicluster-data/ts-2/wal/wals/3e2fa0b0ff444a7fb2e2242d9ce437dd
I20260620 23:11:48.865820 19223 ts_tablet_manager.cc:1939] T 3e2fa0b0ff444a7fb2e2242d9ce437dd P 663f3caaf3a24449ae9075cd10afb8c7: Deleting consensus metadata
I20260620 23:11:48.866811 18590 catalog_manager.cc:4841] TS 663f3caaf3a24449ae9075cd10afb8c7 (127.10.191.195:42731): tablet 3e2fa0b0ff444a7fb2e2242d9ce437dd (table test-table [id=20fdd787278f413789c46eecb47db89b]) successfully deleted
Jun 20 23:11:48 dist-test-slave-jkbq krb5kdc[18561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.191.193@KRBTEST.COM
I20260620 23:11:48.882542 19241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:48.852338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41440 (local address 127.10.191.193:37597)
0620 23:11:48.863326 (+ 10988us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:48.863332 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:48.866678 (+ 3346us) server_negotiation.cc:407] Connection header received
0620 23:11:48.866916 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:48.866924 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:48.867073 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:48.867166 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:48.868736 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:48.869453 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:48.871321 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:48.871770 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:48.871877 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:48.876754 (+ 4877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:48.876769 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:48.876785 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:48.876820 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:48.879764 (+ 2944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:48.880453 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:48.880460 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:48.880462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:48.880584 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:48.880951 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:48.880961 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:48.880963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:48.881314 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:48.881633 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:48.881915 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 23:11:48.882143 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10844,"thread_start_us":10726,"threads_started":1}
I20260620 23:11:48.883424 18808 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:11:49.018817 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 18677
I20260620 23:11:49.041581 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 18817
Jun 20 23:11:49 dist-test-slave-jkbq krb5kdc[18561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.191.194@KRBTEST.COM
I20260620 23:11:49.057021 19231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:49.042523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55930 (local address 127.10.191.194:37179)
0620 23:11:49.042784 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:49.042794 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:49.042814 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:11:49.042925 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:49.042929 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:49.042990 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:49.043117 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:49.044327 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:49.045138 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:49.046396 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:49.046850 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:49.047035 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:49.050975 (+ 3940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:49.050988 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:49.051004 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:49.051038 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:49.054165 (+ 3127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:49.054941 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:49.054948 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:49.054951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:49.055089 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:49.055518 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:49.055524 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:49.055527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:49.055887 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:49.056253 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:49.056445 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 23:11:49.056662 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 23:11:49.057637 18947 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:11:49.207664 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 18817
I20260620 23:11:49.231426 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 18956
Jun 20 23:11:49 dist-test-slave-jkbq krb5kdc[18561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.191.195@KRBTEST.COM
I20260620 23:11:49.251958 19244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:49.232173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52016 (local address 127.10.191.195:42731)
0620 23:11:49.232637 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:49.232641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:49.232659 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:11:49.232714 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:49.232717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:49.232773 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:49.232868 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:49.234183 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:49.235047 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:49.236403 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:49.236780 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:49.236874 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:49.246736 (+ 9862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:49.246754 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:49.246770 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:49.246811 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:49.249637 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:49.250267 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:49.250274 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:49.250276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:49.250391 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:49.250691 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:49.250694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:49.250696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:49.251073 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:49.251288 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:49.251546 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 23:11:49.251703 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":259,"threads_started":1}
I20260620 23:11:49.252590 19086 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:11:49.373611 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 18956
I20260620 23:11:49.394680 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 18574
I20260620 23:11:49.396025 18646 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:11:49.526805 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 18574
2026-06-20T23:11:49Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6304 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997042893763-11007-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:11:49 dist-test-slave-jkbq krb5kdc[19249](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:11:49 dist-test-slave-jkbq krb5kdc[19249](info): set up 2 sockets
Jun 20 23:11:49 dist-test-slave-jkbq krb5kdc[19249](info): commencing operation
krb5kdc: starting...
W20260620 23:11:50.611852 11007 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.006s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Jun 20 23:11:50 dist-test-slave-jkbq krb5kdc[19249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:11:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:11:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.191.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
I20260620 23:11:50.822481 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:36909
--webserver_interface=127.10.191.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:33045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:36909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:11:51.140710 19262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:51.141034 19262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:51.141201 19262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:51.148352 19262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:11:51.148504 19262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:51.148631 19262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:51.148738 19262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:11:51.148820 19262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:11:51.158524 19262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:33045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:36909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:36909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:51.159904 19262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:51.161175 19262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:51.173327 19268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:51.173295 19270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:51.173676 19267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:51.174479 19262 server_base.cc:1034] running on GCE node
I20260620 23:11:51.175318 19262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:51.176659 19262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:51.177899 19262 hybrid_clock.cc:648] HybridClock initialized: now 1781997111177850 us; error 33 us; skew 500 ppm
I20260620 23:11:51.178267 19262 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T23:11:51Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1721 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:11:51 dist-test-slave-jkbq krb5kdc[19276](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:11:51 dist-test-slave-jkbq krb5kdc[19276](info): set up 2 sockets
Jun 20 23:11:51 dist-test-slave-jkbq krb5kdc[19276](info): commencing operation
krb5kdc: starting...
W20260620 23:11:52.351848 11007 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Jun 20 23:11:52 dist-test-slave-jkbq krb5kdc[19276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:11:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:11:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.191.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
I20260620 23:11:52.562188 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:41159
--webserver_interface=127.10.191.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:34851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:41159
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:11:52.874855 19289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:52.875219 19289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:52.875375 19289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:52.882405 19289 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:11:52.882558 19289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:52.882664 19289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:52.882790 19289 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:11:52.882870 19289 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:11:52.892522 19289 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:34851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:41159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:41159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:52.893877 19289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:52.895224 19289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:52.907063 19294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:52.906920 19297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:52.906993 19295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:52.908099 19289 server_base.cc:1034] running on GCE node
I20260620 23:11:52.908854 19289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:52.910220 19289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:52.911443 19289 hybrid_clock.cc:648] HybridClock initialized: now 1781997112911407 us; error 58 us; skew 500 ppm
I20260620 23:11:52.911777 19289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:11:52 dist-test-slave-jkbq krb5kdc[19276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997112, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:52.914762 19289 init.cc:377] Logged in from keytab as kudu/127.10.191.254@KRBTEST.COM (short username kudu)
I20260620 23:11:52.916280 19289 webserver.cc:460] Webserver started at http://127.10.191.254:35437/ using document root <none> and password file <none>
I20260620 23:11:52.916961 19289 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:52.917104 19289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:52.917416 19289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:11:52.920418 19289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "dc37007e18454849a7a6ed6fb803a88b"
format_stamp: "Formatted at 2026-06-20 23:11:52 on dist-test-slave-jkbq"
server_key: "8b060a00d7b61b73635fc0aa796ee0f4"
server_key_iv: "df217ed4057d727563a948f38ba04f91"
server_key_version: "encryptionkey@0"
I20260620 23:11:52.921151 19289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "dc37007e18454849a7a6ed6fb803a88b"
format_stamp: "Formatted at 2026-06-20 23:11:52 on dist-test-slave-jkbq"
server_key: "8b060a00d7b61b73635fc0aa796ee0f4"
server_key_iv: "df217ed4057d727563a948f38ba04f91"
server_key_version: "encryptionkey@0"
I20260620 23:11:52.926157 19289 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 23:11:52.929580 19304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:52.931236 19289 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:11:52.931418 19289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "dc37007e18454849a7a6ed6fb803a88b"
format_stamp: "Formatted at 2026-06-20 23:11:52 on dist-test-slave-jkbq"
server_key: "8b060a00d7b61b73635fc0aa796ee0f4"
server_key_iv: "df217ed4057d727563a948f38ba04f91"
server_key_version: "encryptionkey@0"
I20260620 23:11:52.931617 19289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:52.957710 19289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:52.961828 19289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:52.962117 19289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:52.980170 19289 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:41159
I20260620 23:11:52.980201 19355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:41159 every 8 connection(s)
I20260620 23:11:52.981578 19289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:11:52.985338 19356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:11:52.991613 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 19289
I20260620 23:11:52.991827 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:11:52.992460 11007 external_mini_cluster.cc:1428] Setting key a12c202afd9c31594975ea805344cade
I20260620 23:11:52.995915 19356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc37007e18454849a7a6ed6fb803a88b: Bootstrap starting.
I20260620 23:11:52.999388 19356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc37007e18454849a7a6ed6fb803a88b: Neither blocks nor log segments found. Creating new log.
I20260620 23:11:53.000461 19356 log.cc:824] T 00000000000000000000000000000000 P dc37007e18454849a7a6ed6fb803a88b: Log is configured to *not* fsync() on all Append() calls
I20260620 23:11:53.003545 19356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc37007e18454849a7a6ed6fb803a88b: No bootstrap required, opened a new log
Jun 20 23:11:53 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:11:53.011325 19356 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc37007e18454849a7a6ed6fb803a88b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc37007e18454849a7a6ed6fb803a88b" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 41159 } }
I20260620 23:11:53.011742 19356 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc37007e18454849a7a6ed6fb803a88b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:11:53.011862 19356 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc37007e18454849a7a6ed6fb803a88b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc37007e18454849a7a6ed6fb803a88b, State: Initialized, Role: FOLLOWER
I20260620 23:11:53.012459 19356 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc37007e18454849a7a6ed6fb803a88b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc37007e18454849a7a6ed6fb803a88b" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 41159 } }
I20260620 23:11:53.012723 19356 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc37007e18454849a7a6ed6fb803a88b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:11:53.012874 19356 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc37007e18454849a7a6ed6fb803a88b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:11:53.013074 19356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc37007e18454849a7a6ed6fb803a88b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:11:53.014858 19359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:52.994472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33676 (local address 127.10.191.254:41159)
0620 23:11:52.995277 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:52.995289 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:52.995329 (+ 40us) server_negotiation.cc:407] Connection header received
0620 23:11:52.997044 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:52.997066 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:52.997468 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:52.997781 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:52.999708 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:53.001177 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:53.002940 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:53.003583 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:53.003788 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:53.007586 (+ 3798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:53.007593 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:53.007609 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:53.007653 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:53.010759 (+ 3106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:53.011461 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:53.011468 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:53.011474 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:53.011628 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:53.012060 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:53.012063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:53.012065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:53.012697 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:53.012987 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:53.013410 (+ 423us) server_negotiation.cc:299] Negotiation successful
0620 23:11:53.013818 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":387,"threads_started":1}
I20260620 23:11:53.015632 19356 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc37007e18454849a7a6ed6fb803a88b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc37007e18454849a7a6ed6fb803a88b" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 41159 } }
I20260620 23:11:53.016105 19356 leader_election.cc:302] T 00000000000000000000000000000000 P dc37007e18454849a7a6ed6fb803a88b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc37007e18454849a7a6ed6fb803a88b; no voters:
I20260620 23:11:53.016682 19356 leader_election.cc:288] T 00000000000000000000000000000000 P dc37007e18454849a7a6ed6fb803a88b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:11:53.016865 19361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc37007e18454849a7a6ed6fb803a88b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:11:53.017879 19361 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc37007e18454849a7a6ed6fb803a88b [term 1 LEADER]: Becoming Leader. State: Replica: dc37007e18454849a7a6ed6fb803a88b, State: Running, Role: LEADER
I20260620 23:11:53.018436 19361 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc37007e18454849a7a6ed6fb803a88b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc37007e18454849a7a6ed6fb803a88b" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 41159 } }
I20260620 23:11:53.018847 19356 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc37007e18454849a7a6ed6fb803a88b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:11:53.020968 19363 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc37007e18454849a7a6ed6fb803a88b [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc37007e18454849a7a6ed6fb803a88b. Latest consensus state: current_term: 1 leader_uuid: "dc37007e18454849a7a6ed6fb803a88b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc37007e18454849a7a6ed6fb803a88b" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 41159 } } }
I20260620 23:11:53.021121 19362 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc37007e18454849a7a6ed6fb803a88b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc37007e18454849a7a6ed6fb803a88b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc37007e18454849a7a6ed6fb803a88b" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 41159 } } }
I20260620 23:11:53.021432 19363 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc37007e18454849a7a6ed6fb803a88b [sys.catalog]: This master's current role is: LEADER
I20260620 23:11:53.021504 19362 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc37007e18454849a7a6ed6fb803a88b [sys.catalog]: This master's current role is: LEADER
I20260620 23:11:53.026916 19366 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:11:53.032727 19366 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:11:53.041154 19366 catalog_manager.cc:1347] Generated new cluster ID: dbb74d648ac945e581a644972c53716c
I20260620 23:11:53.041308 19366 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:11:53.063527 19366 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:11:53.065482 19366 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:11:53.073037 19366 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc37007e18454849a7a6ed6fb803a88b: Generated new TSK 0
I20260620 23:11:53.074057 19366 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.191.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab.
Entry for principal kudu/127.10.191.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab.
I20260620 23:11:53.129932 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.193:0
--local_ip_for_outbound_sockets=127.10.191.193
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:41159
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:34851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:11:53.453013 19382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:53.453300 19382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:53.453456 19382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:53.460831 19382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:53.460942 19382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:53.461164 19382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.193
I20260620 23:11:53.470773 19382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:34851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:41159
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.193
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:53.472036 19382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:53.473332 19382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:53.485984 19387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:53.486064 19390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:53.486029 19388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:53.486857 19382 server_base.cc:1034] running on GCE node
I20260620 23:11:53.487489 19382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:53.488363 19382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:53.489552 19382 hybrid_clock.cc:648] HybridClock initialized: now 1781997113489475 us; error 68 us; skew 500 ppm
I20260620 23:11:53.489836 19382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:11:53 dist-test-slave-jkbq krb5kdc[19276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997113, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:53.492955 19382 init.cc:377] Logged in from keytab as kudu/127.10.191.193@KRBTEST.COM (short username kudu)
I20260620 23:11:53.494318 19382 webserver.cc:460] Webserver started at http://127.10.191.193:34047/ using document root <none> and password file <none>
I20260620 23:11:53.495035 19382 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:53.495177 19382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:53.495486 19382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:11:53.498243 19382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-0/data/instance:
uuid: "7a5ce0bfcc5749d3bcbcc70bada0c5de"
format_stamp: "Formatted at 2026-06-20 23:11:53 on dist-test-slave-jkbq"
server_key: "b53cf5aa1b21d6f2ba3d23e652c1c97b"
server_key_iv: "1cfa0a14142548a2051a47fb66246912"
server_key_version: "encryptionkey@0"
I20260620 23:11:53.499143 19382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance:
uuid: "7a5ce0bfcc5749d3bcbcc70bada0c5de"
format_stamp: "Formatted at 2026-06-20 23:11:53 on dist-test-slave-jkbq"
server_key: "b53cf5aa1b21d6f2ba3d23e652c1c97b"
server_key_iv: "1cfa0a14142548a2051a47fb66246912"
server_key_version: "encryptionkey@0"
I20260620 23:11:53.504209 19382 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 23:11:53.507504 19397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:53.508790 19382 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:11:53.508961 19382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-0/wal
uuid: "7a5ce0bfcc5749d3bcbcc70bada0c5de"
format_stamp: "Formatted at 2026-06-20 23:11:53 on dist-test-slave-jkbq"
server_key: "b53cf5aa1b21d6f2ba3d23e652c1c97b"
server_key_iv: "1cfa0a14142548a2051a47fb66246912"
server_key_version: "encryptionkey@0"
I20260620 23:11:53.509176 19382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:53.523598 19382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:53.527769 19382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:53.528038 19382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:53.528954 19382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:11:53.530146 19382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:11:53.530215 19382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:53.530300 19382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:11:53.530345 19382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:53.563854 19382 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.193:34515
I20260620 23:11:53.563874 19509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.193:34515 every 8 connection(s)
I20260620 23:11:53.565131 19382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
I20260620 23:11:53.570241 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 19382
I20260620 23:11:53.570442 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance
I20260620 23:11:53.571252 11007 external_mini_cluster.cc:1428] Setting key 9f16df80310bfcd8901709cc78ebe351
Jun 20 23:11:53 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997113, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:11:53.586436 19512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:53.567783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:48343 (local address 127.10.191.254:41159)
0620 23:11:53.568237 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:53.568244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:53.569194 (+ 950us) server_negotiation.cc:407] Connection header received
0620 23:11:53.571755 (+ 2561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:53.571760 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:53.571831 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:53.571977 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:53.573851 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:53.575053 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:53.576588 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:53.577296 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:53.577438 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:53.580990 (+ 3552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:53.580995 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:53.580998 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:53.581027 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:53.583551 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:53.584243 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:53.584250 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:53.584253 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:53.584383 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:53.584803 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:53.584809 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:53.584813 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:53.585063 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:53.585216 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:53.585871 (+ 655us) server_negotiation.cc:299] Negotiation successful
0620 23:11:53.586079 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":214,"threads_started":1}
I20260620 23:11:53.587468 19513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:53.568352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.193:48343)
0620 23:11:53.569039 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:53.569078 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:53.571381 (+ 2303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:53.572272 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:53.572287 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:53.572979 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:53.573678 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:53.573698 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:53.575202 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:53.575205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:53.576445 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:53.576457 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:53.577491 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:53.577495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:53.577718 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:53.578411 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:53.578433 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:53.580775 (+ 2342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:53.583699 (+ 2924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:53.583709 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:53.583725 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:53.584114 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:53.584503 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:53.584507 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:53.584511 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:53.584694 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:53.585198 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:53.585205 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:53.585522 (+ 317us) client_negotiation.cc:769] Sending connection context
0620 23:11:53.585781 (+ 259us) client_negotiation.cc:240] Negotiation successful
0620 23:11:53.586301 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":283,"threads_started":1}
I20260620 23:11:53.589126 19510 heartbeater.cc:347] Connected to a master server at 127.10.191.254:41159
I20260620 23:11:53.589439 19510 heartbeater.cc:464] Registering TS with master...
I20260620 23:11:53.590165 19510 heartbeater.cc:511] Master 127.10.191.254:41159 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.191.194@KRBTEST.COM; defaulting to no policy
I20260620 23:11:53.592325 19321 ts_manager.cc:194] Registered new tserver with Master: 7a5ce0bfcc5749d3bcbcc70bada0c5de (127.10.191.193:34515)
I20260620 23:11:53.593760 19321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.193@KRBTEST.COM'} at 127.10.191.193:48343
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab.
Entry for principal kudu/127.10.191.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab.
I20260620 23:11:53.618482 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.194:0
--local_ip_for_outbound_sockets=127.10.191.194
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:41159
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:34851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:11:53.934237 19516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:53.934538 19516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:53.934667 19516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:53.941915 19516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:53.942014 19516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:53.942124 19516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.194
I20260620 23:11:53.951615 19516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:34851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:41159
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.194
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:53.952812 19516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:53.954022 19516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:53.966943 19522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:53.966861 19521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:53.967206 19524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:53.968226 19516 server_base.cc:1034] running on GCE node
I20260620 23:11:53.968811 19516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:53.969699 19516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:53.970966 19516 hybrid_clock.cc:648] HybridClock initialized: now 1781997113970867 us; error 77 us; skew 500 ppm
I20260620 23:11:53.971263 19516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:11:53 dist-test-slave-jkbq krb5kdc[19276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997113, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:53.974284 19516 init.cc:377] Logged in from keytab as kudu/127.10.191.194@KRBTEST.COM (short username kudu)
I20260620 23:11:53.975728 19516 webserver.cc:460] Webserver started at http://127.10.191.194:33323/ using document root <none> and password file <none>
I20260620 23:11:53.976449 19516 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:53.976588 19516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:53.976898 19516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:11:53.979823 19516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-1/data/instance:
uuid: "71edbe40429845debb202f7d65d76271"
format_stamp: "Formatted at 2026-06-20 23:11:53 on dist-test-slave-jkbq"
server_key: "4bad0578d7a2e2a656e9650d9f59ae17"
server_key_iv: "dea4a42bb0798a1828973c6417032cae"
server_key_version: "encryptionkey@0"
I20260620 23:11:53.980583 19516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance:
uuid: "71edbe40429845debb202f7d65d76271"
format_stamp: "Formatted at 2026-06-20 23:11:53 on dist-test-slave-jkbq"
server_key: "4bad0578d7a2e2a656e9650d9f59ae17"
server_key_iv: "dea4a42bb0798a1828973c6417032cae"
server_key_version: "encryptionkey@0"
I20260620 23:11:53.985704 19516 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:11:53.989185 19531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:53.990551 19516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 23:11:53.990720 19516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-1/wal
uuid: "71edbe40429845debb202f7d65d76271"
format_stamp: "Formatted at 2026-06-20 23:11:53 on dist-test-slave-jkbq"
server_key: "4bad0578d7a2e2a656e9650d9f59ae17"
server_key_iv: "dea4a42bb0798a1828973c6417032cae"
server_key_version: "encryptionkey@0"
I20260620 23:11:53.990964 19516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:54.010802 19516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:54.015074 19516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:54.015306 19516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:54.016165 19516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:11:54.017431 19516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:11:54.017505 19516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:54.017582 19516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:11:54.017625 19516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:54.050066 19516 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.194:34207
I20260620 23:11:54.050171 19643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.194:34207 every 8 connection(s)
I20260620 23:11:54.051347 19516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
I20260620 23:11:54.058000 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 19516
I20260620 23:11:54.058151 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance
I20260620 23:11:54.058604 11007 external_mini_cluster.cc:1428] Setting key 61872f52fd88c88c7cc34f27b573843d
Jun 20 23:11:54 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997113, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:11:54.071298 19512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:54.053927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:56783 (local address 127.10.191.254:41159)
0620 23:11:54.054124 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:54.054129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:54.055237 (+ 1108us) server_negotiation.cc:407] Connection header received
0620 23:11:54.057029 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:54.057035 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:54.057142 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:54.057256 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:54.058888 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:54.059761 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:54.061462 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:54.061874 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:54.062011 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:54.065404 (+ 3393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:54.065409 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:54.065412 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:54.065442 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:54.068133 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:54.069024 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:54.069028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:54.069030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:54.069152 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:54.069626 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:54.069629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:54.069631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:54.069860 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:54.069962 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:54.070828 (+ 866us) server_negotiation.cc:299] Negotiation successful
0620 23:11:54.071045 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 23:11:54.072806 19646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:54.054343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.194:56783)
0620 23:11:54.055069 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:54.055121 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:54.056767 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:54.057483 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:54.057491 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:54.058011 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:54.058701 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:54.058729 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:54.059909 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:54.059915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:54.061314 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:54.061326 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:54.062107 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:54.062118 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:54.062343 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:54.063077 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:54.063111 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:54.065190 (+ 2079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:54.068270 (+ 3080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:54.068282 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:54.068301 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:54.068918 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:54.069266 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:54.069274 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:54.069280 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:54.069507 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:54.069978 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:54.069989 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:54.070446 (+ 457us) client_negotiation.cc:769] Sending connection context
0620 23:11:54.070775 (+ 329us) client_negotiation.cc:240] Negotiation successful
0620 23:11:54.071365 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":250,"threads_started":1}
I20260620 23:11:54.074306 19644 heartbeater.cc:347] Connected to a master server at 127.10.191.254:41159
I20260620 23:11:54.074599 19644 heartbeater.cc:464] Registering TS with master...
I20260620 23:11:54.075366 19644 heartbeater.cc:511] Master 127.10.191.254:41159 requested a full tablet report, sending...
I20260620 23:11:54.076854 19321 ts_manager.cc:194] Registered new tserver with Master: 71edbe40429845debb202f7d65d76271 (127.10.191.194:34207)
I20260620 23:11:54.077617 19321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.194@KRBTEST.COM'} at 127.10.191.194:56783
WARNING: no policy specified for kudu/127.10.191.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab.
Entry for principal kudu/127.10.191.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab.
I20260620 23:11:54.107115 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.195:0
--local_ip_for_outbound_sockets=127.10.191.195
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:41159
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:34851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:11:54.430794 19649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:54.431137 19649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:54.431319 19649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:54.438648 19649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:54.438802 19649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:54.439044 19649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.195
I20260620 23:11:54.448598 19649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:34851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:41159
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.195
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:54.449928 19649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:54.451285 19649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:54.463980 19654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:54.463985 19655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:54.464286 19657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:54.465062 19649 server_base.cc:1034] running on GCE node
I20260620 23:11:54.465750 19649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:54.466739 19649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:54.467984 19649 hybrid_clock.cc:648] HybridClock initialized: now 1781997114467932 us; error 47 us; skew 500 ppm
I20260620 23:11:54.468319 19649 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:11:54 dist-test-slave-jkbq krb5kdc[19276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997114, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:54.471518 19649 init.cc:377] Logged in from keytab as kudu/127.10.191.195@KRBTEST.COM (short username kudu)
I20260620 23:11:54.472892 19649 webserver.cc:460] Webserver started at http://127.10.191.195:42411/ using document root <none> and password file <none>
I20260620 23:11:54.473536 19649 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:54.473634 19649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:54.473876 19649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:11:54.476732 19649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-2/data/instance:
uuid: "65247ee7813a46f7a82fd9c87b33ab71"
format_stamp: "Formatted at 2026-06-20 23:11:54 on dist-test-slave-jkbq"
server_key: "585e71bfd3a165a7e90edd817b5c9c07"
server_key_iv: "7f3c8e7425e4e00aa175ed1b13f766d4"
server_key_version: "encryptionkey@0"
I20260620 23:11:54.477456 19649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance:
uuid: "65247ee7813a46f7a82fd9c87b33ab71"
format_stamp: "Formatted at 2026-06-20 23:11:54 on dist-test-slave-jkbq"
server_key: "585e71bfd3a165a7e90edd817b5c9c07"
server_key_iv: "7f3c8e7425e4e00aa175ed1b13f766d4"
server_key_version: "encryptionkey@0"
I20260620 23:11:54.482484 19649 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 23:11:54.485944 19664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:54.487329 19649 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:11:54.487497 19649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-2/wal
uuid: "65247ee7813a46f7a82fd9c87b33ab71"
format_stamp: "Formatted at 2026-06-20 23:11:54 on dist-test-slave-jkbq"
server_key: "585e71bfd3a165a7e90edd817b5c9c07"
server_key_iv: "7f3c8e7425e4e00aa175ed1b13f766d4"
server_key_version: "encryptionkey@0"
I20260620 23:11:54.487691 19649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:54.509744 19649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:54.514406 19649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:54.514688 19649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:54.515749 19649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:11:54.517144 19649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:11:54.517269 19649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:54.517418 19649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:11:54.517493 19649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:54.551369 19649 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.195:33551
I20260620 23:11:54.551424 19776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.195:33551 every 8 connection(s)
I20260620 23:11:54.552654 19649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
I20260620 23:11:54.557756 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 19649
I20260620 23:11:54.557936 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance
I20260620 23:11:54.558369 11007 external_mini_cluster.cc:1428] Setting key 72745b95f98b4f8dc324f7ab5176b62d
Jun 20 23:11:54 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997114, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:11:54.572890 19512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:54.555221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:56581 (local address 127.10.191.254:41159)
0620 23:11:54.555391 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:54.555395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:54.556568 (+ 1173us) server_negotiation.cc:407] Connection header received
0620 23:11:54.558475 (+ 1907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:54.558480 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:54.558600 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:54.558711 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:54.560332 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:54.561206 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:54.562717 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:54.563246 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:54.563376 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:54.566937 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:54.566943 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:54.566948 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:54.566988 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:54.569888 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:54.570576 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:54.570580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:54.570582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:54.570723 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:54.571233 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:54.571237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:54.571239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:54.571525 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:54.571641 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:54.572366 (+ 725us) server_negotiation.cc:299] Negotiation successful
0620 23:11:54.572609 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 23:11:54.574002 19779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:54.555753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.195:56581)
0620 23:11:54.556412 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:54.556447 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:54.558215 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:54.559010 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:54.559018 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:54.559624 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:54.560173 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:54.560190 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:54.561355 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:54.561359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:54.562567 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:54.562579 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:54.563463 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:54.563472 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:54.563714 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:54.564395 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:54.564420 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:54.566712 (+ 2292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:11:54.570017 (+ 3305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:54.570024 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:54.570041 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:54.570469 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:54.570849 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:11:54.570853 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:11:54.570856 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:11:54.571124 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:11:54.571670 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:11:54.571678 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:11:54.572024 (+ 346us) client_negotiation.cc:769] Sending connection context
0620 23:11:54.572267 (+ 243us) client_negotiation.cc:240] Negotiation successful
0620 23:11:54.572820 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":255,"threads_started":1}
I20260620 23:11:54.575423 19777 heartbeater.cc:347] Connected to a master server at 127.10.191.254:41159
I20260620 23:11:54.575759 19777 heartbeater.cc:464] Registering TS with master...
I20260620 23:11:54.576454 19777 heartbeater.cc:511] Master 127.10.191.254:41159 requested a full tablet report, sending...
I20260620 23:11:54.577783 19321 ts_manager.cc:194] Registered new tserver with Master: 65247ee7813a46f7a82fd9c87b33ab71 (127.10.191.195:33551)
I20260620 23:11:54.578562 19321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.195@KRBTEST.COM'} at 127.10.191.195:56581
I20260620 23:11:54.585744 11007 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:11:54.596724 19510 heartbeater.cc:503] Master 127.10.191.254:41159 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.10.191.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.191.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.191.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.193.keytab.
Entry for principal oryx/127.10.191.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.193.keytab.
WARNING: no policy specified for oryx/127.10.191.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.191.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.191.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.194.keytab.
Entry for principal oryx/127.10.191.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.194.keytab.
WARNING: no policy specified for oryx/127.10.191.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.191.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.191.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.195.keytab.
Entry for principal oryx/127.10.191.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.195.keytab.
I20260620 23:11:54.711829 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 19382
Jun 20 23:11:54 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.193@KRBTEST.COM
I20260620 23:11:54.727905 19787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:54.713054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50014 (local address 127.10.191.193:34515)
0620 23:11:54.713858 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:54.713867 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:54.713903 (+ 36us) server_negotiation.cc:407] Connection header received
0620 23:11:54.713963 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:54.713969 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:54.714177 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:54.714301 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:54.715647 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:54.716638 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:54.717798 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:54.718339 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:54.718435 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:54.722433 (+ 3998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:54.722443 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:54.722457 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:54.722496 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:54.725357 (+ 2861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:54.725958 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:54.725964 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:54.725967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:54.726084 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:54.726425 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:54.726428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:54.726430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:54.726753 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:54.727057 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:54.727296 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 23:11:54.727485 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":469,"threads_started":1}
I20260620 23:11:54.729125 19505 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:11:54.811338 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 19382
I20260620 23:11:54.827777 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 19516
Jun 20 23:11:54 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.194@KRBTEST.COM
I20260620 23:11:54.843842 19789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:54.828843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53744 (local address 127.10.191.194:34207)
0620 23:11:54.829404 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:54.829411 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:54.829436 (+ 25us) server_negotiation.cc:407] Connection header received
0620 23:11:54.829494 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:54.829500 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:54.829713 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:54.829844 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:54.831130 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:54.832128 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:54.833339 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:54.833901 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:54.834007 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:54.838211 (+ 4204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:54.838222 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:54.838237 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:54.838277 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:54.841076 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:54.841728 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:54.841735 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:54.841738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:54.841858 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:54.842220 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:54.842231 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:54.842233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:54.842571 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:54.842900 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:54.843223 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 23:11:54.843487 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":295,"threads_started":1}
I20260620 23:11:54.845233 19639 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:11:54.926975 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 19516
I20260620 23:11:54.943344 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 19649
Jun 20 23:11:54 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.195@KRBTEST.COM
I20260620 23:11:54.958947 19791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:54.944398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47356 (local address 127.10.191.195:33551)
0620 23:11:54.945002 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:54.945009 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:54.945037 (+ 28us) server_negotiation.cc:407] Connection header received
0620 23:11:54.945096 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:54.945102 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:54.945306 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:54.945434 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:54.946719 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:54.947814 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:54.949023 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:54.949547 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:54.949643 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:54.953969 (+ 4326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:54.953980 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:54.953994 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:54.954028 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:54.956607 (+ 2579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:54.957212 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:54.957219 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:54.957221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:54.957342 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:54.957632 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:54.957635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:54.957637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:54.957962 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:54.958180 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:54.958402 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 23:11:54.958587 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":320,"threads_started":1}
I20260620 23:11:54.960230 19772 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:11:55.043841 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 19649
I20260620 23:11:55.059576 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 19289
I20260620 23:11:55.060773 19351 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:11:55.161888 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 19289
I20260620 23:11:55.182006 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:41159
--webserver_interface=127.10.191.254
--webserver_port=35437
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:34851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:41159
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:11:55.495867 19794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:55.496134 19794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:55.496218 19794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:55.503223 19794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:11:55.503327 19794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:55.503372 19794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:55.503427 19794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:11:55.503470 19794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:11:55.512990 19794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:34851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:41159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:41159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=35437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:55.514209 19794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:55.515453 19794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:55.526770 19799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:55.526732 19802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:55.526798 19800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:55.527283 19794 server_base.cc:1034] running on GCE node
I20260620 23:11:55.528061 19794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:55.529260 19794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:55.530442 19794 hybrid_clock.cc:648] HybridClock initialized: now 1781997115530386 us; error 55 us; skew 500 ppm
I20260620 23:11:55.530707 19794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:11:55 dist-test-slave-jkbq krb5kdc[19276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997115, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:55.533699 19794 init.cc:377] Logged in from keytab as kudu/127.10.191.254@KRBTEST.COM (short username kudu)
I20260620 23:11:55.535161 19794 webserver.cc:460] Webserver started at http://127.10.191.254:35437/ using document root <none> and password file <none>
I20260620 23:11:55.535815 19794 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:55.535970 19794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:55.541116 19794 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260620 23:11:55.543849 19809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:55.545202 19794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 23:11:55.545388 19794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "dc37007e18454849a7a6ed6fb803a88b"
format_stamp: "Formatted at 2026-06-20 23:11:52 on dist-test-slave-jkbq"
server_key: "8b060a00d7b61b73635fc0aa796ee0f4"
server_key_iv: "df217ed4057d727563a948f38ba04f91"
server_key_version: "encryptionkey@0"
I20260620 23:11:55.546710 19794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:55.577976 19794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:55.581755 19794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:55.582010 19794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:55.599920 19794 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:41159
I20260620 23:11:55.599943 19860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:41159 every 8 connection(s)
I20260620 23:11:55.601354 19794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:11:55.602115 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 19794
I20260620 23:11:55.603103 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.193:34515
--local_ip_for_outbound_sockets=127.10.191.193
--tserver_master_addrs=127.10.191.254:41159
--webserver_port=34047
--webserver_interface=127.10.191.193
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:34851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.191.193
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:11:55.608364 19861 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:11:55.611477 19861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc37007e18454849a7a6ed6fb803a88b: Bootstrap starting.
I20260620 23:11:55.629403 19861 log.cc:824] T 00000000000000000000000000000000 P dc37007e18454849a7a6ed6fb803a88b: Log is configured to *not* fsync() on all Append() calls
I20260620 23:11:55.637665 19861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc37007e18454849a7a6ed6fb803a88b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:11:55.638159 19861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc37007e18454849a7a6ed6fb803a88b: Bootstrap complete.
I20260620 23:11:55.646920 19861 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc37007e18454849a7a6ed6fb803a88b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc37007e18454849a7a6ed6fb803a88b" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 41159 } }
I20260620 23:11:55.647774 19861 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc37007e18454849a7a6ed6fb803a88b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc37007e18454849a7a6ed6fb803a88b, State: Initialized, Role: FOLLOWER
I20260620 23:11:55.648430 19861 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc37007e18454849a7a6ed6fb803a88b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc37007e18454849a7a6ed6fb803a88b" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 41159 } }
I20260620 23:11:55.648694 19861 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc37007e18454849a7a6ed6fb803a88b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:11:55.648815 19861 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc37007e18454849a7a6ed6fb803a88b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:11:55.648998 19861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc37007e18454849a7a6ed6fb803a88b [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:11:55.652976 19861 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc37007e18454849a7a6ed6fb803a88b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc37007e18454849a7a6ed6fb803a88b" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 41159 } }
I20260620 23:11:55.653453 19861 leader_election.cc:302] T 00000000000000000000000000000000 P dc37007e18454849a7a6ed6fb803a88b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc37007e18454849a7a6ed6fb803a88b; no voters:
I20260620 23:11:55.654013 19861 leader_election.cc:288] T 00000000000000000000000000000000 P dc37007e18454849a7a6ed6fb803a88b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 23:11:55.654204 19865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc37007e18454849a7a6ed6fb803a88b [term 2 FOLLOWER]: Leader election won for term 2
I20260620 23:11:55.655267 19865 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc37007e18454849a7a6ed6fb803a88b [term 2 LEADER]: Becoming Leader. State: Replica: dc37007e18454849a7a6ed6fb803a88b, State: Running, Role: LEADER
I20260620 23:11:55.655896 19865 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc37007e18454849a7a6ed6fb803a88b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc37007e18454849a7a6ed6fb803a88b" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 41159 } }
I20260620 23:11:55.656018 19861 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc37007e18454849a7a6ed6fb803a88b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:11:55.658474 19867 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc37007e18454849a7a6ed6fb803a88b [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc37007e18454849a7a6ed6fb803a88b. Latest consensus state: current_term: 2 leader_uuid: "dc37007e18454849a7a6ed6fb803a88b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc37007e18454849a7a6ed6fb803a88b" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 41159 } } }
I20260620 23:11:55.658919 19867 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc37007e18454849a7a6ed6fb803a88b [sys.catalog]: This master's current role is: LEADER
I20260620 23:11:55.659533 19866 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc37007e18454849a7a6ed6fb803a88b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "dc37007e18454849a7a6ed6fb803a88b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc37007e18454849a7a6ed6fb803a88b" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 41159 } } }
I20260620 23:11:55.659868 19866 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc37007e18454849a7a6ed6fb803a88b [sys.catalog]: This master's current role is: LEADER
I20260620 23:11:55.664330 19870 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:11:55.669649 19870 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:11:55.671141 19870 catalog_manager.cc:1259] Loaded cluster ID: dbb74d648ac945e581a644972c53716c
I20260620 23:11:55.671285 19870 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:11:55.675308 19870 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:11:55.676725 19870 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dc37007e18454849a7a6ed6fb803a88b: Loaded TSK: 0
I20260620 23:11:55.678211 19870 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 23:11:55.935730 19863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:55.936036 19863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:55.936203 19863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:55.943373 19863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:55.943508 19863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:55.943689 19863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.193
I20260620 23:11:55.953395 19863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:34851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.193.keytab
--principal=oryx/127.10.191.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.193:34515
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.191.193
--webserver_port=34047
--tserver_master_addrs=127.10.191.254:41159
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.193
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:55.954691 19863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:55.956067 19863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:55.968741 19887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:55.968751 19888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:55.969591 19863 server_base.cc:1034] running on GCE node
W20260620 23:11:55.969867 19890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:55.970525 19863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:55.971475 19863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:55.972687 19863 hybrid_clock.cc:648] HybridClock initialized: now 1781997115972639 us; error 42 us; skew 500 ppm
I20260620 23:11:55.973021 19863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:11:55 dist-test-slave-jkbq krb5kdc[19276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997115, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.191.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:55.976150 19863 init.cc:377] Logged in from keytab as oryx/127.10.191.193@KRBTEST.COM (short username oryx)
I20260620 23:11:55.977595 19863 webserver.cc:460] Webserver started at http://127.10.191.193:34047/ using document root <none> and password file <none>
I20260620 23:11:55.978243 19863 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:55.978401 19863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:55.983736 19863 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:11:55.986408 19897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:55.987823 19863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:11:55.987980 19863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-0/wal
uuid: "7a5ce0bfcc5749d3bcbcc70bada0c5de"
format_stamp: "Formatted at 2026-06-20 23:11:53 on dist-test-slave-jkbq"
server_key: "b53cf5aa1b21d6f2ba3d23e652c1c97b"
server_key_iv: "1cfa0a14142548a2051a47fb66246912"
server_key_version: "encryptionkey@0"
I20260620 23:11:55.988883 19863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:56.006660 19863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:56.010618 19863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:56.010912 19863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:56.011857 19863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:11:56.013105 19863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:11:56.013216 19863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:56.013343 19863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:11:56.013444 19863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:56.045957 19863 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.193:34515
I20260620 23:11:56.046051 20009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.193:34515 every 8 connection(s)
I20260620 23:11:56.047417 19863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
I20260620 23:11:56.054266 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 19863
I20260620 23:11:56.055047 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.194:34207
--local_ip_for_outbound_sockets=127.10.191.194
--tserver_master_addrs=127.10.191.254:41159
--webserver_port=33323
--webserver_interface=127.10.191.194
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:34851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.191.194
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:11:56 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.193@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:11:56 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.193@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:11:56.065282 20013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:56.050483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.193:49847)
0620 23:11:56.051270 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:56.051320 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:56.052958 (+ 1638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:56.054462 (+ 1504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:56.054470 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:56.055008 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:56.055607 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:56.055623 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:56.057259 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:56.057263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:56.058472 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:56.058484 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:56.059550 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:56.059557 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:56.059830 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:56.060483 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:56.060505 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:56.064017 (+ 3512us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":349,"threads_started":1}
W20260620 23:11:56.065313 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:11:56.050307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:49847 (local address 127.10.191.254:41159)
0620 23:11:56.051028 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:56.051039 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:56.051487 (+ 448us) server_negotiation.cc:407] Connection header received
0620 23:11:56.053424 (+ 1937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:56.053436 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:56.053876 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:56.054314 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:56.055838 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:56.057089 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:56.058619 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:56.059328 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:56.059519 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:56.064355 (+ 4836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.193:49847: BlockingRecv error: recv got EOF from 127.10.191.193:49847 (error 108)
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":274,"threads_started":1}
W20260620 23:11:56.066092 20010 heartbeater.cc:644] Failed to heartbeat to 127.10.191.254:41159 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.191.254:41159: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Jun 20 23:11:56 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.193@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:11:56 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.193@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:11:56.074802 20013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:56.066682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.193:38277)
0620 23:11:56.066852 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:56.066914 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:56.067002 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:56.067338 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:56.067341 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:56.067611 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:56.067741 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:56.067759 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:56.069241 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:56.069245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:56.070103 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:56.070112 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:56.071167 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:56.071171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:56.071287 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:56.071851 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:56.071867 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:56.074614 (+ 2747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 23:11:56.075500 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:11:56.066766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:38277 (local address 127.10.191.254:41159)
0620 23:11:56.066967 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:56.066975 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:56.066996 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:11:56.067114 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:56.067122 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:56.067206 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:56.067314 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:56.067889 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:56.069108 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:56.070237 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:56.071033 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:56.071178 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:56.074928 (+ 3750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.193:38277: BlockingRecv error: recv got EOF from 127.10.191.193:38277 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 23:11:56 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.193@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:11:56 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.193@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:11:56.084281 20013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:56.075503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.193:44239)
0620 23:11:56.075736 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:56.075754 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:56.075847 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:56.076407 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:56.076415 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:56.076813 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:56.076986 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:56.076995 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:56.077918 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:56.077923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:56.079400 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:56.079413 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:56.080108 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:56.080114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:56.080308 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:56.080877 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:56.080898 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:56.084010 (+ 3112us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131}
W20260620 23:11:56.084383 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:11:56.075724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:44239 (local address 127.10.191.254:41159)
0620 23:11:56.075902 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:56.075913 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:56.075934 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:11:56.075987 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:56.075992 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:56.076100 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:56.076210 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:56.077117 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:56.077788 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:56.079533 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:56.079969 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:56.080069 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:56.084159 (+ 4090us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.193:44239: BlockingRecv error: recv got EOF from 127.10.191.193:44239 (error 108)
Metrics: {"server-negotiator.queue_time_us":59,"thread_start_us":427,"threads_started":1}
W20260620 23:11:56.084600 20010 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 23:11:56.374931 20014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:56.375272 20014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:56.375443 20014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:56.382562 20014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:56.382719 20014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:56.382925 20014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.194
I20260620 23:11:56.392625 20014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:34851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.194.keytab
--principal=oryx/127.10.191.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.194:34207
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.191.194
--webserver_port=33323
--tserver_master_addrs=127.10.191.254:41159
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.194
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:56.393985 20014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:56.395468 20014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:56.408454 20020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:56.408425 20021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:56.408653 20023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:56.409200 20014 server_base.cc:1034] running on GCE node
I20260620 23:11:56.409729 20014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:56.410560 20014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:56.411756 20014 hybrid_clock.cc:648] HybridClock initialized: now 1781997116411718 us; error 29 us; skew 500 ppm
I20260620 23:11:56.412075 20014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:11:56 dist-test-slave-jkbq krb5kdc[19276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997116, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.191.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:56.418219 20014 init.cc:377] Logged in from keytab as oryx/127.10.191.194@KRBTEST.COM (short username oryx)
I20260620 23:11:56.419711 20014 webserver.cc:460] Webserver started at http://127.10.191.194:33323/ using document root <none> and password file <none>
I20260620 23:11:56.420369 20014 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:56.420508 20014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:56.425676 20014 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 23:11:56.428370 20030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:56.429637 20014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:11:56.429803 20014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-1/wal
uuid: "71edbe40429845debb202f7d65d76271"
format_stamp: "Formatted at 2026-06-20 23:11:53 on dist-test-slave-jkbq"
server_key: "4bad0578d7a2e2a656e9650d9f59ae17"
server_key_iv: "dea4a42bb0798a1828973c6417032cae"
server_key_version: "encryptionkey@0"
I20260620 23:11:56.430684 20014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:56.454264 20014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:56.458186 20014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:56.458456 20014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:56.459440 20014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:11:56.460708 20014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:11:56.460794 20014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:56.460922 20014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:11:56.461019 20014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:56.493893 20014 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.194:34207
I20260620 23:11:56.493952 20142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.194:34207 every 8 connection(s)
I20260620 23:11:56.495433 20014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
I20260620 23:11:56.495985 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 20014
I20260620 23:11:56.496747 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.195:33551
--local_ip_for_outbound_sockets=127.10.191.195
--tserver_master_addrs=127.10.191.254:41159
--webserver_port=42411
--webserver_interface=127.10.191.195
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:34851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.191.195
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:11:56 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.194@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:11:56 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.194@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 23:11:56.511117 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:11:56.498335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:59465 (local address 127.10.191.254:41159)
0620 23:11:56.498571 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:56.498580 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:56.499815 (+ 1235us) server_negotiation.cc:407] Connection header received
0620 23:11:56.501649 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:56.501658 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:56.501752 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:56.501875 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:56.503377 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:56.504084 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:56.505581 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:56.506086 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:56.506211 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:56.510848 (+ 4637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.194:59465: BlockingRecv error: recv got EOF from 127.10.191.194:59465 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 23:11:56.511782 20145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:56.498805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.194:59465)
0620 23:11:56.499647 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:56.499691 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:56.501383 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:56.502059 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:56.502066 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:56.502572 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:56.503210 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:56.503228 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:56.504246 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:56.504251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:56.505439 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:56.505450 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:56.506306 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:56.506312 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:56.506533 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:56.507207 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:56.507227 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:56.510689 (+ 3462us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":324,"threads_started":1}
W20260620 23:11:56.512553 20143 heartbeater.cc:644] Failed to heartbeat to 127.10.191.254:41159 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.191.254:41159: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Jun 20 23:11:56 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.194@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:11:56 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.194@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:11:56.521806 20145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:56.513099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.194:45103)
0620 23:11:56.513313 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:56.513334 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:56.513444 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:56.513745 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:56.513753 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:56.514206 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:56.514421 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:56.514434 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:56.515416 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:56.515424 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:56.517133 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:56.517152 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:56.517823 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:56.517832 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:56.518055 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:56.518646 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:56.518673 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:56.521635 (+ 2962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":69}
W20260620 23:11:56.522324 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:11:56.513252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:45103 (local address 127.10.191.254:41159)
0620 23:11:56.513401 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:56.513405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:56.513420 (+ 15us) server_negotiation.cc:407] Connection header received
0620 23:11:56.513550 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:56.513553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:56.513605 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:56.513682 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:56.514566 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:56.515287 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:56.517274 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:56.517673 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:56.517768 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:56.521850 (+ 4082us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.194:45103: BlockingRecv error: recv got EOF from 127.10.191.194:45103 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 23:11:56 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.194@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:11:56 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.194@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:11:56.529897 20145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:56.522393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.194:41311)
0620 23:11:56.522635 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:56.522655 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:56.522744 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:56.523199 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:56.523204 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:56.523468 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:56.523596 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:56.523603 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:56.524528 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:56.524533 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:56.525394 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:56.525403 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:56.526060 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:56.526066 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:56.526231 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:56.526761 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:56.526776 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:56.529620 (+ 2844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109}
W20260620 23:11:56.530012 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:11:56.522647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:41311 (local address 127.10.191.254:41159)
0620 23:11:56.522971 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:56.522975 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:56.522988 (+ 13us) server_negotiation.cc:407] Connection header received
0620 23:11:56.523021 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:56.523024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:56.523070 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:56.523182 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:56.523733 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:56.524396 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:56.525527 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:56.525928 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:56.526052 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:56.529815 (+ 3763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.194:41311: BlockingRecv error: recv got EOF from 127.10.191.194:41311 (error 108)
Metrics: {"server-negotiator.queue_time_us":177}
W20260620 23:11:56.530236 20143 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 23:11:56.817943 20146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:11:56.818205 20146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:11:56.818296 20146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:11:56.825239 20146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:11:56.825336 20146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:11:56.825443 20146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.195
I20260620 23:11:56.834710 20146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:34851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/krb5kdc/oryx_127.10.191.195.keytab
--principal=oryx/127.10.191.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.195:33551
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.191.195
--webserver_port=42411
--tserver_master_addrs=127.10.191.254:41159
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.195
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:11:56.835983 20146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:11:56.837240 20146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:11:56.849749 20154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:56.849861 20152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:11:56.849861 20151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:11:56.850684 20146 server_base.cc:1034] running on GCE node
I20260620 23:11:56.851327 20146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:11:56.852267 20146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:11:56.853456 20146 hybrid_clock.cc:648] HybridClock initialized: now 1781997116853421 us; error 30 us; skew 500 ppm
I20260620 23:11:56.853757 20146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:11:56 dist-test-slave-jkbq krb5kdc[19276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997116, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.191.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:11:56.856792 20146 init.cc:377] Logged in from keytab as oryx/127.10.191.195@KRBTEST.COM (short username oryx)
I20260620 23:11:56.858178 20146 webserver.cc:460] Webserver started at http://127.10.191.195:42411/ using document root <none> and password file <none>
I20260620 23:11:56.858826 20146 fs_manager.cc:359] Metadata directory not provided
I20260620 23:11:56.859009 20146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:11:56.864003 20146 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260620 23:11:56.866770 20161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:56.868165 20146 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:11:56.868371 20146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-2/wal
uuid: "65247ee7813a46f7a82fd9c87b33ab71"
format_stamp: "Formatted at 2026-06-20 23:11:54 on dist-test-slave-jkbq"
server_key: "585e71bfd3a165a7e90edd817b5c9c07"
server_key_iv: "7f3c8e7425e4e00aa175ed1b13f766d4"
server_key_version: "encryptionkey@0"
I20260620 23:11:56.869385 20146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:11:56.883198 20146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:11:56.887313 20146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:11:56.887583 20146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:11:56.888520 20146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:11:56.889818 20146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:11:56.889936 20146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:56.890072 20146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:11:56.890168 20146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:11:56.923257 20146 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.195:33551
I20260620 23:11:56.923341 20273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.195:33551 every 8 connection(s)
I20260620 23:11:56.924818 20146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
I20260620 23:11:56.928001 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 20146
Jun 20 23:11:56 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.195@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:11:56 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.195@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 23:11:56.941998 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:11:56.927263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:35919 (local address 127.10.191.254:41159)
0620 23:11:56.927433 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:56.927438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:56.928766 (+ 1328us) server_negotiation.cc:407] Connection header received
0620 23:11:56.931547 (+ 2781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:56.931551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:56.931615 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:56.931694 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:56.933348 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:56.934126 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:56.936305 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:56.936720 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:56.936858 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:56.941797 (+ 4939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.195:35919: BlockingRecv error: recv got EOF from 127.10.191.195:35919 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 23:11:56.942937 20276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:56.927783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.195:35919)
0620 23:11:56.928610 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:56.928655 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:56.931273 (+ 2618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:56.931955 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:56.931963 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:56.932440 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:56.933179 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:56.933201 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:56.934298 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:56.934302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:56.936154 (+ 1852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:56.936170 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:56.936948 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:56.936956 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:56.937269 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:56.938018 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:56.938038 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:56.941630 (+ 3592us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":335,"threads_started":1}
W20260620 23:11:56.943521 20274 heartbeater.cc:644] Failed to heartbeat to 127.10.191.254:41159 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.191.254:41159: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
I20260620 23:11:56.944955 20278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:56.928936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36010 (local address 127.10.191.254:41159)
0620 23:11:56.929671 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:56.929675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:56.929693 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:11:56.929742 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:56.929745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:56.929802 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:56.929891 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:56.931873 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:56.932580 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:56.934116 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:56.934582 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:56.934680 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:56.937925 (+ 3245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:11:56.937941 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:11:56.937965 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:11:56.938004 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:11:56.941422 (+ 3418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:56.942262 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:56.942279 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:56.942286 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:56.942440 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:11:56.942920 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:11:56.942926 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:11:56.942930 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:11:56.943574 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:11:56.943884 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 23:11:56.944302 (+ 418us) server_negotiation.cc:299] Negotiation successful
0620 23:11:56.944655 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":539,"threads_started":1}
Jun 20 23:11:56 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.195@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:11:56 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.195@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:11:56.952463 20276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:56.944106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.195:38753)
0620 23:11:56.944271 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:56.944289 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:56.944380 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:56.944734 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:56.944738 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:56.945985 (+ 1247us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:56.946129 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:56.946137 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:56.947348 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:56.947354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:56.948223 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:56.948234 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:56.948928 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:56.948933 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:56.949099 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:56.949546 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:56.949561 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:56.952197 (+ 2636us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 23:11:56.952587 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:11:56.944345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:38753 (local address 127.10.191.254:41159)
0620 23:11:56.944497 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:56.944510 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:56.944523 (+ 13us) server_negotiation.cc:407] Connection header received
0620 23:11:56.944556 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:56.944558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:56.944604 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:56.944712 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:56.946269 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:56.947221 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:56.948367 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:56.948787 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:56.948919 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:56.952360 (+ 3441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.195:38753: BlockingRecv error: recv got EOF from 127.10.191.195:38753 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 23:11:56 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.195@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:11:56 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.195@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:11:56.960631 20276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:56.953134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.195:36609)
0620 23:11:56.953298 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:56.953312 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:56.953415 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:56.953763 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:56.953768 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:56.954040 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:56.954178 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:56.954186 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:56.955194 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:56.955201 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:56.956152 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:56.956163 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:56.956887 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:56.956893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:56.957120 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:56.957586 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:56.957601 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:56.960442 (+ 2841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 23:11:56.960812 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:11:56.953258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:36609 (local address 127.10.191.254:41159)
0620 23:11:56.953489 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:56.953496 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:56.953512 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:11:56.953556 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:56.953559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:56.953642 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:56.953741 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:56.954303 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:56.955063 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:56.956277 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:56.956747 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:56.956867 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:56.960588 (+ 3721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.195:36609: BlockingRecv error: recv got EOF from 127.10.191.195:36609 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
W20260620 23:11:56.960949 20274 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 23:11:57 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.193@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:11:57 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.193@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:11:57.093374 20279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:57.085146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.193:45603)
0620 23:11:57.085635 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:57.085651 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:57.085775 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:57.086061 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:57.086065 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:57.086505 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:57.086710 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:57.086720 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:57.087727 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:57.087731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:57.088708 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:57.088719 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:57.089395 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:57.089400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:57.089519 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:57.090225 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:57.090242 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:57.093109 (+ 2867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":249,"threads_started":1}
W20260620 23:11:57.093493 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:11:57.085304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:45603 (local address 127.10.191.254:41159)
0620 23:11:57.085524 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:57.085538 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:57.085726 (+ 188us) server_negotiation.cc:407] Connection header received
0620 23:11:57.085875 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:57.085879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:57.085938 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:57.086017 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:57.086828 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:57.087606 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:57.088825 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:57.089242 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:57.089332 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:57.093311 (+ 3979us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.193:45603: BlockingRecv error: recv got EOF from 127.10.191.193:45603 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 20 23:11:57 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.194@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:11:57 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.194@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:11:57.539011 20280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:57.530789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.194:52445)
0620 23:11:57.531403 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:57.531419 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:57.531524 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:57.531804 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:57.531807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:57.532230 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:57.532427 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:57.532436 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:57.533388 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:57.533392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:57.534376 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:57.534417 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:57.535207 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:57.535212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:57.535331 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:57.535874 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:57.535891 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:57.538727 (+ 2836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":386,"threads_started":1}
W20260620 23:11:57.539193 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:11:57.531225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:52445 (local address 127.10.191.254:41159)
0620 23:11:57.531406 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:57.531411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:57.531468 (+ 57us) server_negotiation.cc:407] Connection header received
0620 23:11:57.531617 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:57.531620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:57.531680 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:57.531755 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:57.532552 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:57.533252 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:57.534599 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:57.535063 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:57.535155 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:57.538957 (+ 3802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.194:52445: BlockingRecv error: recv got EOF from 127.10.191.194:52445 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 23:11:57 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.195@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:11:57 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.195@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:11:57.969928 20281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:57.961570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.195:54595)
0620 23:11:57.962085 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:57.962101 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:57.962219 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:57.962511 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:57.962514 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:57.963041 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:57.963239 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:57.963248 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:57.964279 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:57.964283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:57.965319 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:57.965331 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:57.966197 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:57.966203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:57.966379 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:57.967071 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:57.967088 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:57.969708 (+ 2620us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":283,"threads_started":1}
W20260620 23:11:57.970110 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:11:57.961683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:54595 (local address 127.10.191.254:41159)
0620 23:11:57.961856 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:57.961861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:57.962200 (+ 339us) server_negotiation.cc:407] Connection header received
0620 23:11:57.962321 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:57.962325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:57.962387 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:57.962472 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:57.963383 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:57.964155 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:57.965465 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:57.966013 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:57.966150 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:57.969881 (+ 3731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.195:54595: BlockingRecv error: recv got EOF from 127.10.191.195:54595 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 23:11:58 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.193@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:11:58 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.193@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:11:58.102834 20282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:58.094267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.193:39841)
0620 23:11:58.094789 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:58.094817 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:58.094960 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:58.095267 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:58.095270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:58.095645 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:58.095814 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:58.095825 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:58.097031 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:58.097040 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:58.098120 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:58.098132 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:58.098995 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:58.099007 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:58.099126 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:58.099681 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:58.099698 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:58.102426 (+ 2728us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":325,"threads_started":1}
W20260620 23:11:58.102893 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:11:58.094362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:39841 (local address 127.10.191.254:41159)
0620 23:11:58.094538 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:58.094542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:58.094890 (+ 348us) server_negotiation.cc:407] Connection header received
0620 23:11:58.095062 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:58.095065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:58.095130 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:58.095232 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:58.095937 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:58.096887 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:58.098267 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:58.098804 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:58.098980 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:58.102634 (+ 3654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.193:39841: BlockingRecv error: recv got EOF from 127.10.191.193:39841 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 23:11:58 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.194@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:11:58 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.194@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:11:58.547927 20283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:58.539882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.194:42029)
0620 23:11:58.540469 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:58.540485 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:58.540599 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:58.540895 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:58.540897 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:58.541319 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:58.541491 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:58.541499 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:58.542502 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:58.542506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:58.543420 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:58.543431 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:58.544072 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:58.544076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:58.544191 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:58.544774 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:58.544791 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:58.547567 (+ 2776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":324,"threads_started":1}
W20260620 23:11:58.547973 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:11:58.539944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:42029 (local address 127.10.191.254:41159)
0620 23:11:58.540098 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:58.540102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:58.540563 (+ 461us) server_negotiation.cc:407] Connection header received
0620 23:11:58.540694 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:58.540698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:58.540769 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:58.540852 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:58.541657 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:58.542387 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:58.543555 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:58.543948 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:58.544037 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:58.547764 (+ 3727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.194:42029: BlockingRecv error: recv got EOF from 127.10.191.194:42029 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 23:11:58 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.195@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:11:58 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.195@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:11:58.979163 20284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:58.970785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.195:40925)
0620 23:11:58.971347 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:58.971363 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:58.971516 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:58.971872 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:58.971877 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:58.972279 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:58.972489 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:58.972507 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:58.973573 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:58.973581 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:58.974558 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:58.974569 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:58.975351 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:58.975356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:58.975520 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:58.976137 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:58.976155 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:58.978795 (+ 2640us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":293,"threads_started":1}
W20260620 23:11:58.979277 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:11:58.971043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:40925 (local address 127.10.191.254:41159)
0620 23:11:58.971252 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:58.971259 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:58.971442 (+ 183us) server_negotiation.cc:407] Connection header received
0620 23:11:58.971634 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:58.971639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:58.971724 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:58.971838 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:58.972629 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:58.973422 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:58.974698 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:58.975205 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:58.975337 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:58.979019 (+ 3682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.195:40925: BlockingRecv error: recv got EOF from 127.10.191.195:40925 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 23:11:59 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.193@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:11:59 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.193@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:11:59.112653 20285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:59.103919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.193:44007)
0620 23:11:59.104417 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:59.104432 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:59.104590 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:59.104938 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:59.104943 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:59.105312 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:59.105468 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:59.105476 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:59.106609 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:59.106613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:59.108015 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:59.108032 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:59.108801 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:59.108807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:59.108931 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:59.109509 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:59.109526 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:59.112391 (+ 2865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":275,"threads_started":1}
W20260620 23:11:59.112824 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:11:59.103997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:44007 (local address 127.10.191.254:41159)
0620 23:11:59.104230 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:59.104239 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:59.104522 (+ 283us) server_negotiation.cc:407] Connection header received
0620 23:11:59.104724 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:59.104735 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:59.104804 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:59.104906 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:59.105620 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:59.106424 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:59.108186 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:59.108661 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:59.108762 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:59.112580 (+ 3818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.193:44007: BlockingRecv error: recv got EOF from 127.10.191.193:44007 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Jun 20 23:11:59 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.194@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:11:59 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.194@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:11:59.558846 20286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:59.548996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.194:33147)
0620 23:11:59.549444 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:59.549459 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:59.549603 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:59.549999 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:59.550004 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:59.550481 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:59.550740 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:59.550772 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:59.552499 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:59.552506 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:59.553398 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:59.553410 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:59.554291 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:59.554297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:59.554498 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:59.555154 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:59.555190 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:59.558400 (+ 3210us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":241,"threads_started":1}
W20260620 23:11:59.558867 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:11:59.549037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:33147 (local address 127.10.191.254:41159)
0620 23:11:59.549239 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:59.549247 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:59.549537 (+ 290us) server_negotiation.cc:407] Connection header received
0620 23:11:59.549738 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:59.549745 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:59.549839 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:59.549948 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:59.550938 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:59.552348 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:59.553538 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:59.554139 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:59.554274 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:59.558600 (+ 4326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.194:33147: BlockingRecv error: recv got EOF from 127.10.191.194:33147 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 23:11:59 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.195@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:11:59 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.195@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:11:59.987870 20287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:11:59.980054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.195:36395)
0620 23:11:59.980535 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0620 23:11:59.980555 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:11:59.980668 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:11:59.981021 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:11:59.981032 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:11:59.981373 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:11:59.981533 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:59.981545 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:59.982520 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:59.982535 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:59.983456 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:11:59.983468 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:11:59.984149 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:11:59.984154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:11:59.984293 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:59.984863 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:11:59.984880 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:11:59.987613 (+ 2733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":240,"threads_started":1}
W20260620 23:11:59.988039 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:11:59.980195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:36395 (local address 127.10.191.254:41159)
0620 23:11:59.980419 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 23:11:59.980424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:11:59.980630 (+ 206us) server_negotiation.cc:407] Connection header received
0620 23:11:59.980791 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:11:59.980798 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:11:59.980898 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:11:59.980976 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:11:59.981661 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:59.982376 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:59.983586 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:11:59.983990 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:11:59.984081 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:11:59.987831 (+ 3750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.195:36395: BlockingRecv error: recv got EOF from 127.10.191.195:36395 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 20 23:12:00 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.193@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:12:00 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.193@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:12:00.121656 20288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:00.113514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.193:55965)
0620 23:12:00.114019 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:00.114035 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:00.114144 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:00.114549 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:00.114555 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:00.114966 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:00.115112 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:00.115121 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:00.116050 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:00.116053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:00.116991 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:00.117001 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:00.117707 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:00.117711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:00.117823 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:00.118444 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:00.118462 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:00.121440 (+ 2978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":288,"threads_started":1}
W20260620 23:12:00.121978 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:12:00.113675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:55965 (local address 127.10.191.254:41159)
0620 23:12:00.113859 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:00.113863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:00.114119 (+ 256us) server_negotiation.cc:407] Connection header received
0620 23:12:00.114283 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:00.114287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:00.114349 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:00.114503 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:00.115237 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:00.115942 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:00.117125 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:00.117581 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:00.117691 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:00.121706 (+ 4015us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.193:55965: BlockingRecv error: recv got EOF from 127.10.191.193:55965 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 20 23:12:00 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.194@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:12:00 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.194@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:12:00.569200 20289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:00.559926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.194:55731)
0620 23:12:00.560471 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:00.560490 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:00.560610 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:00.560901 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:00.560904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:00.561261 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:00.561413 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:00.561421 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:00.562412 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:00.562415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:00.563921 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:00.563935 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:00.565112 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:00.565120 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:00.565340 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:00.566252 (+ 912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:00.566277 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:00.568968 (+ 2691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":340,"threads_started":1}
W20260620 23:12:00.569358 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:12:00.559966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:55731 (local address 127.10.191.254:41159)
0620 23:12:00.560163 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:00.560172 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:00.560573 (+ 401us) server_negotiation.cc:407] Connection header received
0620 23:12:00.560707 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:00.560710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:00.560782 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:00.560862 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:00.561579 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:00.562298 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:00.564065 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:00.564735 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:00.564868 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:00.569141 (+ 4273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.194:55731: BlockingRecv error: recv got EOF from 127.10.191.194:55731 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 23:12:00 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.195@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:12:00 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.195@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:12:00.997403 20290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:00.988752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.195:39675)
0620 23:12:00.989458 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:00.989475 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:00.989610 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:00.989891 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:00.989894 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:00.990242 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:00.990388 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:00.990396 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:00.991582 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:00.991586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:00.992577 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:00.992589 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:00.993361 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:00.993367 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:00.993531 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:00.994283 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:00.994300 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:00.997102 (+ 2802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":493,"threads_started":1}
W20260620 23:12:00.997614 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:12:00.988829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:39675 (local address 127.10.191.254:41159)
0620 23:12:00.989072 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:00.989082 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:00.989551 (+ 469us) server_negotiation.cc:407] Connection header received
0620 23:12:00.989708 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:00.989712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:00.989771 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:00.989850 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:00.990535 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:00.991431 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:00.992799 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:00.993204 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:00.993336 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:00.997334 (+ 3998us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.195:39675: BlockingRecv error: recv got EOF from 127.10.191.195:39675 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 20 23:12:01 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.193@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:12:01 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.193@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:12:01.131168 20291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:01.122627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.193:33263)
0620 23:12:01.123184 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:01.123200 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:01.123353 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:01.123702 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:01.123707 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:01.124164 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:01.124324 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:01.124333 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:01.125370 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:01.125377 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:01.126424 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:01.126437 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:01.127212 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:01.127218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:01.127398 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:01.128039 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:01.128057 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:01.130856 (+ 2799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":278,"threads_started":1}
W20260620 23:12:01.131223 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:12:01.122696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:33263 (local address 127.10.191.254:41159)
0620 23:12:01.122968 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:01.122977 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:01.123282 (+ 305us) server_negotiation.cc:407] Connection header received
0620 23:12:01.123480 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:01.123486 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:01.123557 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:01.123670 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:01.124479 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:01.125236 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:01.126561 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:01.127071 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:01.127202 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:01.131065 (+ 3863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.193:33263: BlockingRecv error: recv got EOF from 127.10.191.193:33263 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 23:12:01 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.194@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:12:01 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.194@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:12:01.583531 20292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:01.570255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.194:51015)
0620 23:12:01.570796 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:01.570811 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:01.570983 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:01.571327 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:01.571332 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:01.571736 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:01.571888 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:01.571897 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:01.572914 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:01.572919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:01.574100 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:01.574112 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:01.574796 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:01.574802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:01.575005 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:01.580461 (+ 5456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:01.580507 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:01.583293 (+ 2786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":266,"threads_started":1}
W20260620 23:12:01.583726 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:12:01.570423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:51015 (local address 127.10.191.254:41159)
0620 23:12:01.570648 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:01.570654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:01.570904 (+ 250us) server_negotiation.cc:407] Connection header received
0620 23:12:01.571127 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:01.571132 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:01.571203 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:01.571301 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:01.572027 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:01.572753 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:01.574243 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:01.574657 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:01.574784 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:01.583468 (+ 8684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.194:51015: BlockingRecv error: recv got EOF from 127.10.191.194:51015 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 20 23:12:02 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.195@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:12:02 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.195@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:12:02.006618 20293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:01.998505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.195:57629)
0620 23:12:01.999045 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:01.999063 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:01.999169 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:01.999449 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:01.999452 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:01.999847 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:02.000001 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:02.000009 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:02.001008 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:02.001014 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:02.002048 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:02.002061 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:02.002726 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:02.002731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:02.002845 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:02.003462 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:02.003480 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:02.006303 (+ 2823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":257,"threads_started":1}
W20260620 23:12:02.006891 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:12:01.998569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:57629 (local address 127.10.191.254:41159)
0620 23:12:01.998762 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:01.998766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:01.999134 (+ 368us) server_negotiation.cc:407] Connection header received
0620 23:12:01.999268 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:01.999271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:01.999329 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:01.999411 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:02.000127 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:02.000855 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:02.002178 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:02.002599 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:02.002692 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:02.006534 (+ 3842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.195:57629: BlockingRecv error: recv got EOF from 127.10.191.195:57629 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 20 23:12:02 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.193@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:12:02 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.193@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:12:02.140324 20294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:02.132152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.193:46711)
0620 23:12:02.132721 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:02.132736 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:02.132848 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:02.133121 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:02.133124 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:02.133499 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:02.133718 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:02.133727 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:02.134716 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:02.134722 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:02.135716 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:02.135726 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:02.136368 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:02.136372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:02.136489 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:02.137091 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:02.137107 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:02.140125 (+ 3018us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":325,"threads_started":1}
W20260620 23:12:02.140538 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:12:02.132208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:46711 (local address 127.10.191.254:41159)
0620 23:12:02.132382 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:02.132386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:02.132807 (+ 421us) server_negotiation.cc:407] Connection header received
0620 23:12:02.132943 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:02.132947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:02.133003 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:02.133082 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:02.133836 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:02.134562 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:02.135842 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:02.136237 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:02.136327 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:02.140288 (+ 3961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.193:46711: BlockingRecv error: recv got EOF from 127.10.191.193:46711 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 23:12:02 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.194@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:12:02 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.194@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:12:02.592473 20295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:02.584439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.194:59181)
0620 23:12:02.584904 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:02.584920 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:02.585030 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:02.585307 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:02.585311 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:02.585651 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:02.585888 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:02.585897 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:02.586928 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:02.586939 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:02.587805 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:02.587815 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:02.588483 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:02.588487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:02.588601 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:02.589298 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:02.589315 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:02.592140 (+ 2825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":265,"threads_started":1}
W20260620 23:12:02.592640 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:12:02.584517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:59181 (local address 127.10.191.254:41159)
0620 23:12:02.584770 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:02.584779 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:02.585004 (+ 225us) server_negotiation.cc:407] Connection header received
0620 23:12:02.585122 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:02.585126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:02.585189 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:02.585268 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:02.586035 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:02.586766 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:02.587929 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:02.588313 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:02.588465 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:02.592381 (+ 3916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.194:59181: BlockingRecv error: recv got EOF from 127.10.191.194:59181 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Jun 20 23:12:03 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.195@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:12:03 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.195@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:12:03.015724 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:03.007537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.195:37857)
0620 23:12:03.008096 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:03.008119 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:03.008226 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:03.008506 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:03.008509 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:03.008868 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:03.009023 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:03.009030 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:03.010169 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:03.010173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:03.011152 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:03.011164 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:03.011886 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:03.011890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:03.012007 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:03.012648 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:03.012665 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:03.015416 (+ 2751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":323,"threads_started":1}
W20260620 23:12:03.015841 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:12:03.007576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:37857 (local address 127.10.191.254:41159)
0620 23:12:03.007772 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:03.007777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:03.008191 (+ 414us) server_negotiation.cc:407] Connection header received
0620 23:12:03.008319 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:03.008323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:03.008382 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:03.008464 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:03.009171 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:03.010036 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:03.011291 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:03.011754 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:03.011848 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:03.015587 (+ 3739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.195:37857: BlockingRecv error: recv got EOF from 127.10.191.195:37857 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jun 20 23:12:03 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.193@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:12:03 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.193@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:12:03.149789 20297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:03.141258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.193:50105)
0620 23:12:03.141866 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:03.141882 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:03.141999 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:03.142280 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:03.142283 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:03.142702 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:03.142891 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:03.142900 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:03.143903 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:03.143909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:03.144886 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:03.144897 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:03.145627 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:03.145633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:03.145805 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:03.146345 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:03.146362 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:03.149405 (+ 3043us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":287,"threads_started":1}
W20260620 23:12:03.149828 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:12:03.141330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:50105 (local address 127.10.191.254:41159)
0620 23:12:03.141508 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:03.141512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:03.141949 (+ 437us) server_negotiation.cc:407] Connection header received
0620 23:12:03.142093 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:03.142096 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:03.142154 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:03.142243 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:03.143040 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:03.143762 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:03.145038 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:03.145474 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:03.145599 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:03.149599 (+ 4000us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.193:50105: BlockingRecv error: recv got EOF from 127.10.191.193:50105 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 23:12:03 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.194@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:12:03 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.194@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:12:03.601758 20298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:03.593276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.194:60663)
0620 23:12:03.593908 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:03.593923 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:03.594082 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:03.594436 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:03.594441 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:03.594888 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:03.595053 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:03.595062 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:03.596089 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:03.596096 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:03.597009 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:03.597020 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:03.597740 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:03.597746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:03.597918 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:03.598446 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:03.598463 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:03.601427 (+ 2964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":302,"threads_started":1}
W20260620 23:12:03.601800 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:12:03.593426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:60663 (local address 127.10.191.254:41159)
0620 23:12:03.593721 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:03.593730 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:03.594013 (+ 283us) server_negotiation.cc:407] Connection header received
0620 23:12:03.594234 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:03.594237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:03.594298 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:03.594400 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:03.595220 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:03.595956 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:03.597162 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:03.597586 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:03.597725 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:03.601603 (+ 3878us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.194:60663: BlockingRecv error: recv got EOF from 127.10.191.194:60663 (error 108)
Metrics: {"server-negotiator.queue_time_us":129}
Jun 20 23:12:04 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.195@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:12:04 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.195@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:12:04.024650 20299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:04.016819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.195:54827)
0620 23:12:04.017369 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:04.017392 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:04.017505 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:04.017839 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:04.017842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:04.018225 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:04.018400 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:04.018409 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:04.019343 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:04.019347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:04.020261 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:04.020272 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:04.020938 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:04.020942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:04.021049 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:04.021580 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:04.021598 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:04.024353 (+ 2755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":258,"threads_started":1}
W20260620 23:12:04.024803 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:12:04.016932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:54827 (local address 127.10.191.254:41159)
0620 23:12:04.017088 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:04.017093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:04.017462 (+ 369us) server_negotiation.cc:407] Connection header received
0620 23:12:04.017642 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:04.017646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:04.017708 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:04.017796 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:04.018514 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:04.019234 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:04.020392 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:04.020820 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:04.020908 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:04.024601 (+ 3693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.195:54827: BlockingRecv error: recv got EOF from 127.10.191.195:54827 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 23:12:04 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.193@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:12:04 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.193@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:12:04.159529 20300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:04.150875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.193:45025)
0620 23:12:04.151430 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:04.151445 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:04.151589 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:04.151918 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:04.151921 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:04.152294 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:04.152439 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:04.152446 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:04.153617 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:04.153621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:04.154580 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:04.154592 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:04.155692 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:04.155698 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:04.155887 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:04.156630 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:04.156647 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:04.159307 (+ 2660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":283,"threads_started":1}
W20260620 23:12:04.159699 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:12:04.151024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:45025 (local address 127.10.191.254:41159)
0620 23:12:04.151231 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:04.151236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:04.151537 (+ 301us) server_negotiation.cc:407] Connection header received
0620 23:12:04.151729 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:04.151733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:04.151791 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:04.151876 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:04.152559 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:04.153449 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:04.154739 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:04.155510 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:04.155623 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:04.159473 (+ 3850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.193:45025: BlockingRecv error: recv got EOF from 127.10.191.193:45025 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 23:12:04 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.194@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:12:04 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.194@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:12:04.611645 20301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:04.602713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.194:43341)
0620 23:12:04.603417 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:04.603434 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:04.603604 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:04.603961 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:04.603966 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:04.604380 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:04.604529 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:04.604540 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:04.605769 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:04.605774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:04.606704 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:04.606716 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:04.607551 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:04.607559 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:04.607714 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:04.608286 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:04.608305 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:04.611317 (+ 3012us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":469,"threads_started":1}
W20260620 23:12:04.611790 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:12:04.602843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:43341 (local address 127.10.191.254:41159)
0620 23:12:04.603076 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:04.603081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:04.603567 (+ 486us) server_negotiation.cc:407] Connection header received
0620 23:12:04.603744 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:04.603749 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:04.603825 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:04.603931 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:04.604678 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:04.605608 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:04.606858 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:04.607346 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:04.607483 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:04.611539 (+ 4056us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.194:43341: BlockingRecv error: recv got EOF from 127.10.191.194:43341 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Jun 20 23:12:05 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.195@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:12:05 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.195@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:12:05.034044 20302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:05.025626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.195:49747)
0620 23:12:05.026168 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:05.026184 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:05.026340 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:05.026715 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:05.026720 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:05.027181 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:05.027331 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:05.027340 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:05.028334 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:05.028339 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:05.029254 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:05.029265 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:05.029993 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:05.030006 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:05.030161 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:05.030759 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:05.030775 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:05.033668 (+ 2893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":293,"threads_started":1}
W20260620 23:12:05.034080 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:12:05.025706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:49747 (local address 127.10.191.254:41159)
0620 23:12:05.025952 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:05.025961 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:05.026267 (+ 306us) server_negotiation.cc:407] Connection header received
0620 23:12:05.026469 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:05.026474 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:05.026558 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:05.026684 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:05.027467 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:05.028186 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:05.029392 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:05.029818 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:05.029956 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:05.033857 (+ 3901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.195:49747: BlockingRecv error: recv got EOF from 127.10.191.195:49747 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 20 23:12:05 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.193@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:12:05 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.193@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:12:05.168727 20303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:05.160454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.193:38795)
0620 23:12:05.160993 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:05.161010 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:05.161173 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:05.161517 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:05.161521 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:05.161931 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:05.162104 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:05.162113 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:05.163412 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:05.163417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:05.164361 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:05.164374 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:05.165044 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:05.165048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:05.165177 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:05.165715 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:05.165732 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:05.168494 (+ 2762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":305,"threads_started":1}
W20260620 23:12:05.168936 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:12:05.160508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:38795 (local address 127.10.191.254:41159)
0620 23:12:05.160749 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:05.160759 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:05.161097 (+ 338us) server_negotiation.cc:407] Connection header received
0620 23:12:05.161317 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:05.161322 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:05.161381 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:05.161483 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:05.162246 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:05.163205 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:05.164490 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:05.164913 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:05.165009 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:05.168669 (+ 3660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.193:38795: BlockingRecv error: recv got EOF from 127.10.191.193:38795 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 23:12:05 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.194@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:12:05 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.194@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:12:05.620908 20304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:05.612778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.194:39755)
0620 23:12:05.613340 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:05.613356 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:05.613464 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:05.613864 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:05.613869 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:05.614435 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:05.614614 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:05.614623 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:05.615591 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:05.615595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:05.616595 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:05.616607 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:05.617254 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:05.617258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:05.617372 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:05.617988 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:05.618005 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:05.620679 (+ 2674us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":310,"threads_started":1}
W20260620 23:12:05.621076 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:12:05.612861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:39755 (local address 127.10.191.254:41159)
0620 23:12:05.613063 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:05.613070 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:05.613441 (+ 371us) server_negotiation.cc:407] Connection header received
0620 23:12:05.613604 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:05.613611 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:05.613687 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:05.613798 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:05.614747 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:05.615476 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:05.616725 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:05.617120 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:05.617216 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:05.620845 (+ 3629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.194:39755: BlockingRecv error: recv got EOF from 127.10.191.194:39755 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 20 23:12:06 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.195@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:12:06 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.195@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:12:06.043269 20305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:06.034957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.195:34555)
0620 23:12:06.035533 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:06.035549 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:06.035658 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:06.035933 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:06.035936 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:06.036311 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:06.036471 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:06.036479 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:06.037401 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:06.037404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:06.038307 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:06.038317 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:06.039002 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:06.039006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:06.039127 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:06.039754 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:06.039771 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:06.042941 (+ 3170us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":315,"threads_started":1}
W20260620 23:12:06.043413 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:12:06.034994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:34555 (local address 127.10.191.254:41159)
0620 23:12:06.035149 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:06.035154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:06.035622 (+ 468us) server_negotiation.cc:407] Connection header received
0620 23:12:06.035751 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:06.035754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:06.035812 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:06.035891 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:06.036587 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:06.037292 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:06.038422 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:06.038819 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:06.038979 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:06.043161 (+ 4182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.195:34555: BlockingRecv error: recv got EOF from 127.10.191.195:34555 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 23:12:06 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.193@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:12:06 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.193@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:12:06.178112 20306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:06.169642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.193:33525)
0620 23:12:06.170117 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:06.170133 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:06.170246 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:06.170574 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:06.170577 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:06.170959 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:06.171100 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:06.171108 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:06.172121 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:06.172126 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:06.173482 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:06.173496 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:06.174308 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:06.174313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:06.174426 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:06.175025 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:06.175041 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:06.177782 (+ 2741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":282,"threads_started":1}
W20260620 23:12:06.178207 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:12:06.169816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:33525 (local address 127.10.191.254:41159)
0620 23:12:06.170060 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:06.170066 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:06.170212 (+ 146us) server_negotiation.cc:407] Connection header received
0620 23:12:06.170345 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:06.170349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:06.170407 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:06.170486 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:06.171216 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:06.171968 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:06.173629 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:06.174173 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:06.174272 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:06.177966 (+ 3694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.193:33525: BlockingRecv error: recv got EOF from 127.10.191.193:33525 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Jun 20 23:12:06 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.194@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:12:06 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.194@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:12:06.634553 20307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:06.621874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.194:44449)
0620 23:12:06.622387 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:06.622404 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:06.622530 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:06.622847 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:06.622851 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:06.623289 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:06.623433 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:06.623441 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:06.624466 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:06.624470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:06.625403 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:06.625416 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:06.626312 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:06.626317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:06.626452 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:06.631438 (+ 4986us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:06.631458 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:06.634224 (+ 2766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":298,"threads_started":1}
W20260620 23:12:06.634688 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:12:06.621936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:44449 (local address 127.10.191.254:41159)
0620 23:12:06.622110 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:06.622115 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:06.622495 (+ 380us) server_negotiation.cc:407] Connection header received
0620 23:12:06.622636 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:06.622640 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:06.622699 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:06.622784 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:06.623577 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:06.624316 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:06.625539 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:06.626131 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:06.626227 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:06.634415 (+ 8188us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.194:44449: BlockingRecv error: recv got EOF from 127.10.191.194:44449 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 23:12:07 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.195@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:12:07 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.195@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:12:07.052352 20308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:07.044165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.195:33747)
0620 23:12:07.044719 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:07.044739 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:07.044886 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:07.045211 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:07.045216 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:07.045613 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:07.045761 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:07.045769 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:07.046784 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:07.046793 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:07.047752 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:07.047763 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:07.048468 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:07.048473 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:07.048644 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:07.049281 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:07.049298 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:07.052133 (+ 2835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":272,"threads_started":1}
W20260620 23:12:07.052547 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:12:07.044365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:33747 (local address 127.10.191.254:41159)
0620 23:12:07.044568 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:07.044574 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:07.044819 (+ 245us) server_negotiation.cc:407] Connection header received
0620 23:12:07.045000 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:07.045006 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:07.045076 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:07.045179 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:07.045898 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:07.046654 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:07.047893 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:07.048319 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:07.048455 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:07.052310 (+ 3855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.195:33747: BlockingRecv error: recv got EOF from 127.10.191.195:33747 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 20 23:12:07 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.193@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:12:07 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.193@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:12:07.188199 20309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:07.178996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.193:60509)
0620 23:12:07.179558 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:07.179575 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:07.179742 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:07.180149 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:07.180156 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:07.180669 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:07.180853 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:07.180862 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:07.182002 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:07.182009 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:07.183006 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:07.183017 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:07.183810 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:07.183816 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:07.184069 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:07.184681 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:07.184699 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:07.187874 (+ 3175us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":293,"threads_started":1}
W20260620 23:12:07.188375 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:12:07.179086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:60509 (local address 127.10.191.254:41159)
0620 23:12:07.179418 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:07.179425 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:07.179673 (+ 248us) server_negotiation.cc:407] Connection header received
0620 23:12:07.179895 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:07.179901 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:07.179975 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:07.180109 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:07.181007 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:07.181852 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:07.183165 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:07.183647 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:07.183798 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:07.188112 (+ 4314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.193:60509: BlockingRecv error: recv got EOF from 127.10.191.193:60509 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Jun 20 23:12:07 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.194@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:12:07 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.194@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:12:07.643826 20310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:07.635643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.194:52285)
0620 23:12:07.636117 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:07.636131 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:07.636292 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:07.636624 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:07.636629 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:07.637063 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:07.637231 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:07.637240 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:07.638395 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:07.638403 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:07.639310 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:07.639321 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:07.640068 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:07.640074 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:07.640261 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:07.640803 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:07.640821 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:07.643585 (+ 2764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":259,"threads_started":1}
W20260620 23:12:07.644021 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:12:07.635720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:52285 (local address 127.10.191.254:41159)
0620 23:12:07.635947 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:07.635956 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:07.636223 (+ 267us) server_negotiation.cc:407] Connection header received
0620 23:12:07.636417 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:07.636423 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:07.636493 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:07.636594 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:07.637378 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:07.638231 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:07.639467 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:07.639920 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:07.640057 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:07.643773 (+ 3716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.194:52285: BlockingRecv error: recv got EOF from 127.10.191.194:52285 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 20 23:12:08 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.195@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:12:08 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.195@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:12:08.061769 20311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:08.053225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.195:51405)
0620 23:12:08.053800 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:08.053815 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:08.053931 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:08.054243 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:08.054246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:08.054624 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:08.054817 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:08.054825 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:08.056062 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:08.056068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:08.057179 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:08.057192 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:08.057896 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:08.057902 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:08.058093 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:08.058659 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:08.058677 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:08.061509 (+ 2832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":335,"threads_started":1}
W20260620 23:12:08.061862 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:12:08.053344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:51405 (local address 127.10.191.254:41159)
0620 23:12:08.053537 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:08.053541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:08.053914 (+ 373us) server_negotiation.cc:407] Connection header received
0620 23:12:08.054042 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:08.054046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:08.054108 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:08.054196 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:08.055019 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:08.055879 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:08.057315 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:08.057734 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:08.057872 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:08.061684 (+ 3812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.195:51405: BlockingRecv error: recv got EOF from 127.10.191.195:51405 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 20 23:12:08 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.193@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:12:08 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.193@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 23:12:08.198493 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:12:08.189271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:60041 (local address 127.10.191.254:41159)
0620 23:12:08.189456 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:08.189462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:08.189930 (+ 468us) server_negotiation.cc:407] Connection header received
0620 23:12:08.190149 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:08.190154 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:08.190235 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:08.190338 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:08.191194 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:08.192019 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:08.193454 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:08.193967 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:08.194115 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:08.198307 (+ 4192us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.193:60041: BlockingRecv error: recv got EOF from 127.10.191.193:60041 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 23:12:08.198590 20312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:08.189197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.193:60041)
0620 23:12:08.189825 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:08.189840 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:08.190021 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:08.190365 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:08.190369 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:08.190815 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:08.191060 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:08.191068 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:08.192170 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:08.192186 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:08.193319 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:08.193331 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:08.194141 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:08.194147 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:08.194346 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:08.195028 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:08.195046 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:08.198150 (+ 3104us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":395,"threads_started":1}
Jun 20 23:12:08 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.194@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:12:08 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.194@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:12:08.653228 20313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:08.644739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.194:33799)
0620 23:12:08.645312 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:08.645328 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:08.645482 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:08.645850 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:08.645855 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:08.646262 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:08.646412 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:08.646420 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:08.647553 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:08.647559 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:08.648463 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:08.648474 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:08.649182 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:08.649187 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:08.649372 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:08.649966 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:08.649984 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:08.652989 (+ 3005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":328,"threads_started":1}
W20260620 23:12:08.653430 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:12:08.644881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:33799 (local address 127.10.191.254:41159)
0620 23:12:08.645078 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:08.645085 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:08.645414 (+ 329us) server_negotiation.cc:407] Connection header received
0620 23:12:08.645625 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:08.645632 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:08.645702 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:08.645807 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:08.646545 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:08.647412 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:08.648612 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:08.649041 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:08.649173 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:08.653188 (+ 4015us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.194:33799: BlockingRecv error: recv got EOF from 127.10.191.194:33799 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 23:12:09 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.195@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:12:09 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.195@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:12:09.071233 20314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:09.062937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.195:34541)
0620 23:12:09.063503 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:09.063522 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:09.063673 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:09.064114 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:09.064118 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:09.064604 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:09.064751 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:09.064761 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:09.065732 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:09.065735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:09.066631 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:09.066641 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:09.067333 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:09.067339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:09.067540 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:09.068112 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:09.068128 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:09.070872 (+ 2744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":314,"threads_started":1}
W20260620 23:12:09.071333 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:12:09.062944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:34541 (local address 127.10.191.254:41159)
0620 23:12:09.063146 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:09.063154 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:09.063611 (+ 457us) server_negotiation.cc:407] Connection header received
0620 23:12:09.063836 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:09.063840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:09.063936 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:09.064043 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:09.064880 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:09.065613 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:09.066745 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:09.067186 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:09.067274 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:09.071098 (+ 3824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.195:34541: BlockingRecv error: recv got EOF from 127.10.191.195:34541 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 23:12:09 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.193@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:12:09 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.193@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:12:09.208294 20315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:09.199609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.193:49777)
0620 23:12:09.200157 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:09.200172 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:09.200329 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:09.200716 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:09.200721 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:09.201106 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:09.201257 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:09.201265 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:09.202311 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:09.202318 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:09.203270 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:09.203281 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:09.204026 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:09.204032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:09.204210 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:09.204783 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:09.204800 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:09.207935 (+ 3135us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":294,"threads_started":1}
W20260620 23:12:09.208374 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:12:09.199697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:49777 (local address 127.10.191.254:41159)
0620 23:12:09.199966 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:09.199976 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:09.200261 (+ 285us) server_negotiation.cc:407] Connection header received
0620 23:12:09.200468 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:09.200473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:09.200559 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:09.200660 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:09.201396 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:09.202144 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:09.203423 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:09.203860 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:09.204031 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:09.208132 (+ 4101us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.193:49777: BlockingRecv error: recv got EOF from 127.10.191.193:49777 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
Jun 20 23:12:09 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.194@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:12:09 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.194@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:12:09.662492 20316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:09.654145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.194:55623)
0620 23:12:09.654700 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:09.654716 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:09.654834 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:09.655122 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:09.655126 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:09.655485 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:09.655653 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:09.655661 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:09.656728 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:09.656733 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:09.657827 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:09.657840 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:09.658519 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:09.658525 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:09.658717 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:09.659403 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:09.659420 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:09.662260 (+ 2840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":299,"threads_started":1}
W20260620 23:12:09.662717 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:12:09.654394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:55623 (local address 127.10.191.254:41159)
0620 23:12:09.654555 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:09.654559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:09.654788 (+ 229us) server_negotiation.cc:407] Connection header received
0620 23:12:09.654934 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:09.654937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:09.654996 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:09.655075 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:09.655815 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:09.656574 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:09.657961 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:09.658350 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:09.658437 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:09.662482 (+ 4045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.194:55623: BlockingRecv error: recv got EOF from 127.10.191.194:55623 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 23:12:10 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.195@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:12:10 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.195@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:12:10.080590 20317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:10.072230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.195:42445)
0620 23:12:10.072784 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:10.072800 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:10.072912 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:10.073231 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:10.073238 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:10.073613 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:10.073792 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:10.073805 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:10.074810 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:10.074814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:10.075800 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:10.075812 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:10.076545 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:10.076550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:10.076670 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:10.077298 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:10.077316 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:10.080294 (+ 2978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":300,"threads_started":1}
W20260620 23:12:10.080739 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:12:10.072310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:42445 (local address 127.10.191.254:41159)
0620 23:12:10.072498 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:10.072505 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:10.072870 (+ 365us) server_negotiation.cc:407] Connection header received
0620 23:12:10.073007 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:10.073011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:10.073071 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:10.073162 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:10.073944 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:10.074674 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:10.075969 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:10.076376 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:10.076478 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:10.080478 (+ 4000us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.195:42445: BlockingRecv error: recv got EOF from 127.10.191.195:42445 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 23:12:10 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.193@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:12:10 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.193@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:12:10.217195 20318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:10.209274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.193:43601)
0620 23:12:10.209807 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:10.209830 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:10.209949 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:10.210259 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:10.210262 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:10.210596 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:10.210764 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:10.210772 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:10.211795 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:10.211799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:10.212734 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:10.212746 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:10.213432 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:10.213437 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:10.213579 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:10.214176 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:10.214195 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:10.216951 (+ 2756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":267,"threads_started":1}
W20260620 23:12:10.217375 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:12:10.209480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:43601 (local address 127.10.191.254:41159)
0620 23:12:10.209739 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:10.209747 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:10.209896 (+ 149us) server_negotiation.cc:407] Connection header received
0620 23:12:10.210054 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:10.210057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:10.210137 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:10.210216 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:10.210914 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:10.211659 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:10.212869 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:10.213283 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:10.213376 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:10.217157 (+ 3781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.193:43601: BlockingRecv error: recv got EOF from 127.10.191.193:43601 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
Jun 20 23:12:10 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.194@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:12:10 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.194@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:12:10.671355 20319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:10.663456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.194:60659)
0620 23:12:10.663934 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:10.663952 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:10.664063 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:10.664359 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:10.664362 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:10.664715 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:10.664881 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:10.664889 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:10.665887 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:10.665891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:10.666780 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:10.666790 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:10.667515 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:10.667518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:10.667637 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:10.668197 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:10.668215 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:10.671114 (+ 2899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":273,"threads_started":1}
W20260620 23:12:10.671559 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:12:10.663600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:60659 (local address 127.10.191.254:41159)
0620 23:12:10.663771 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:10.663775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:10.664057 (+ 282us) server_negotiation.cc:407] Connection header received
0620 23:12:10.664162 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:10.664166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:10.664231 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:10.664316 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:10.665017 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:10.665765 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:10.666928 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:10.667381 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:10.667488 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:10.671315 (+ 3827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.194:60659: BlockingRecv error: recv got EOF from 127.10.191.194:60659 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 23:12:11 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.195@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:12:11 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.195@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:12:11.089996 20320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:11.081623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.195:52399)
0620 23:12:11.082061 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:11.082078 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:11.082212 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:11.082535 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:11.082539 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:11.082969 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:11.083154 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:11.083163 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:11.084110 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:11.084113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:11.085026 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:11.085038 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:11.085876 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:11.085882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:11.086060 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:11.086905 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:11.086931 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:11.089751 (+ 2820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":240,"threads_started":1}
W20260620 23:12:11.090190 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:12:11.081813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:52399 (local address 127.10.191.254:41159)
0620 23:12:11.082043 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:11.082049 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:11.082166 (+ 117us) server_negotiation.cc:407] Connection header received
0620 23:12:11.082306 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:11.082310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:11.082368 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:11.082454 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:11.083288 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:11.083985 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:11.085169 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:11.085708 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:11.085839 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:11.089951 (+ 4112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.195:52399: BlockingRecv error: recv got EOF from 127.10.191.195:52399 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Jun 20 23:12:11 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.193@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:12:11 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.193@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:12:11.226728 20321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:11.218458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.193:33941)
0620 23:12:11.219141 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:11.219157 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:11.219312 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:11.219709 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:11.219713 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:11.220096 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:11.220256 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:11.220265 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:11.221308 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:11.221313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:11.222260 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:11.222270 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:11.223109 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:11.223115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:11.223291 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:11.223824 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:11.223843 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:11.226497 (+ 2654us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":443,"threads_started":1}
W20260620 23:12:11.227108 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:12:11.218501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:33941 (local address 127.10.191.254:41159)
0620 23:12:11.218724 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:11.218734 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:11.219243 (+ 509us) server_negotiation.cc:407] Connection header received
0620 23:12:11.219437 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:11.219442 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:11.219532 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:11.219640 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:11.220397 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:11.221159 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:11.222402 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:11.222950 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:11.223077 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:11.226739 (+ 3662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.193:33941: BlockingRecv error: recv got EOF from 127.10.191.193:33941 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 20 23:12:11 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.194@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:12:11 dist-test-slave-jkbq krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.191.194@KRBTEST.COM for oryx/127.10.191.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:12:11.686334 20322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:11.672545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:41159 (local address 127.10.191.194:40717)
0620 23:12:11.673166 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:11.673181 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:11.673353 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:11.673734 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:11.673738 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:11.674297 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:11.674542 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:11.674556 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:11.675746 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:11.675750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:11.676670 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:11.676682 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:11.677554 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:11.677562 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:11.677714 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:11.683031 (+ 5317us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:11.683055 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:11.685919 (+ 2864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.191.254:41159: Server oryx/127.10.191.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":331,"threads_started":1}
W20260620 23:12:11.686434 20012 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:12:11.672647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:40717 (local address 127.10.191.254:41159)
0620 23:12:11.672847 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:11.672856 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:11.673256 (+ 400us) server_negotiation.cc:407] Connection header received
0620 23:12:11.673486 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:11.673491 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:11.673568 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:11.673686 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:11.674690 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:11.675615 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:11.676839 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:11.677324 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:11.677514 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:11.686162 (+ 8648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.191.194:40717: BlockingRecv error: recv got EOF from 127.10.191.194:40717 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 23:12:11.934913 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 19863
I20260620 23:12:11.946730 20324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:11.935934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49078 (local address 127.10.191.193:34515)
0620 23:12:11.936526 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:11.936534 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:11.936559 (+ 25us) server_negotiation.cc:407] Connection header received
0620 23:12:11.936617 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:11.936624 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:11.936851 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:11.937002 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:11.938328 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:11.939431 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:11.940602 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:11.941108 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:11.941204 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:11.943659 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:11.943668 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:11.943682 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:11.943713 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:11.946254 (+ 2541us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.10.191.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 23:12:11.946509 (+ 255us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49078: Request ticket server kudu/127.10.191.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":318,"threads_started":1}
W20260620 23:12:11.946864 20324 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49078: Request ticket server kudu/127.10.191.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 23:12:11.946892 11007 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 19863: Not authorized: Client connection negotiation failed: client connection to 127.10.191.193:34515: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.10.191.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 23:12:11.947108 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 19863
I20260620 23:12:11.964123 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 20014
I20260620 23:12:11.976226 20325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:11.965142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53408 (local address 127.10.191.194:34207)
0620 23:12:11.965713 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:11.965720 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:11.965748 (+ 28us) server_negotiation.cc:407] Connection header received
0620 23:12:11.965810 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:11.965816 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:11.966022 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:11.966189 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:11.967570 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:11.968720 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:11.969965 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:11.970524 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:11.970637 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:11.973178 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:11.973189 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:11.973202 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:11.973237 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:11.975707 (+ 2470us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.10.191.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 23:12:11.975941 (+ 234us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53408: Request ticket server kudu/127.10.191.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":263,"threads_started":1}
W20260620 23:12:11.976325 20325 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53408: Request ticket server kudu/127.10.191.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 23:12:11.976359 11007 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 20014: Not authorized: Client connection negotiation failed: client connection to 127.10.191.194:34207: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.10.191.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 23:12:11.976517 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 20014
I20260620 23:12:11.993181 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 20146
I20260620 23:12:12.005131 20326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:11.994130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49434 (local address 127.10.191.195:33551)
0620 23:12:11.994676 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:11.994683 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:11.994707 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:12:11.994765 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:11.994771 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:11.995060 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:11.995263 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:11.996474 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:11.997423 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:11.998647 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:11.999265 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:11.999405 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:12.001891 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:12.001899 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:12.001913 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:12.001942 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:12.004414 (+ 2472us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.10.191.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 23:12:12.004782 (+ 368us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49434: Request ticket server kudu/127.10.191.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":238,"threads_started":1}
W20260620 23:12:12.005180 11007 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 20146: Not authorized: Client connection negotiation failed: client connection to 127.10.191.195:33551: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.10.191.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 23:12:12.005244 20326 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49434: Request ticket server kudu/127.10.191.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 23:12:12.005347 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 20146
I20260620 23:12:12.018076 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 19794
I20260620 23:12:12.019428 19856 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:12:12.132565 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 19794
2026-06-20T23:12:12Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20885 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T23:12:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:12:12Z Disabled control of system clock
I20260620 23:12:12.206188 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:39169
--webserver_interface=127.10.191.254
--webserver_port=0
--builtin_ntp_servers=127.10.191.212:35899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:39169
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 23:12:12.522786 20331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:12:12.523152 20331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:12:12.523238 20331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:12:12.530243 20331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:12:12.530342 20331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:12:12.530381 20331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:12:12.530431 20331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:12:12.530470 20331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:12:12.539423 20331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:35899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:39169
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:39169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:12:12.540619 20331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:12:12.542297 20331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:12:12.553821 20336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:12.553889 20337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:12.553862 20339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:12:12.555011 20331 server_base.cc:1034] running on GCE node
I20260620 23:12:12.555842 20331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:12:12.557271 20331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:12:12.558512 20331 hybrid_clock.cc:648] HybridClock initialized: now 1781997132558480 us; error 61 us; skew 500 ppm
I20260620 23:12:12.558933 20331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:12:12.561509 20331 webserver.cc:460] Webserver started at http://127.10.191.254:45167/ using document root <none> and password file <none>
I20260620 23:12:12.562230 20331 fs_manager.cc:359] Metadata directory not provided
I20260620 23:12:12.562364 20331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:12:12.562716 20331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:12:12.565696 20331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "2555c2746fd546f4ab74cc32bc72af6c"
format_stamp: "Formatted at 2026-06-20 23:12:12 on dist-test-slave-jkbq"
server_key: "96d687b745087aa8355a3ddaf394de74"
server_key_iv: "baf3631ce7a80d5079afe27ebfd083d3"
server_key_version: "encryptionkey@0"
I20260620 23:12:12.566460 20331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "2555c2746fd546f4ab74cc32bc72af6c"
format_stamp: "Formatted at 2026-06-20 23:12:12 on dist-test-slave-jkbq"
server_key: "96d687b745087aa8355a3ddaf394de74"
server_key_iv: "baf3631ce7a80d5079afe27ebfd083d3"
server_key_version: "encryptionkey@0"
I20260620 23:12:12.571547 20331 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 23:12:12.575045 20345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:12.576365 20331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:12:12.576550 20331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "2555c2746fd546f4ab74cc32bc72af6c"
format_stamp: "Formatted at 2026-06-20 23:12:12 on dist-test-slave-jkbq"
server_key: "96d687b745087aa8355a3ddaf394de74"
server_key_iv: "baf3631ce7a80d5079afe27ebfd083d3"
server_key_version: "encryptionkey@0"
I20260620 23:12:12.576776 20331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:12:12.588651 20331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:12:12.589435 20331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:12:12.589720 20331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:12:12.609413 20331 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:39169
I20260620 23:12:12.609433 20396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:39169 every 8 connection(s)
I20260620 23:12:12.610780 20331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:12:12.614698 20397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:12:12.614987 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 20331
I20260620 23:12:12.615188 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:12:12.616015 11007 external_mini_cluster.cc:1428] Setting key bcfcad9d6f2250821f7017f0d9bef45e
I20260620 23:12:12.628791 20397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2555c2746fd546f4ab74cc32bc72af6c: Bootstrap starting.
I20260620 23:12:12.630231 20400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:12.618189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51444 (local address 127.10.191.254:39169)
0620 23:12:12.619263 (+ 1074us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:12.619274 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:12.619317 (+ 43us) server_negotiation.cc:407] Connection header received
0620 23:12:12.621141 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:12.621164 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:12.621628 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:12.621974 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:12.622619 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:12.623917 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:12.625808 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:12.626507 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:12.626636 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:12.627941 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:12.627948 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:12.627962 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:12:12.628492 (+ 530us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:12.628568 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:12:12.628588 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:12.628678 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:12.628942 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 23:12:12.629290 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":410,"threads_started":1}
I20260620 23:12:12.632196 20397 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2555c2746fd546f4ab74cc32bc72af6c: Neither blocks nor log segments found. Creating new log.
I20260620 23:12:12.633348 20397 log.cc:824] T 00000000000000000000000000000000 P 2555c2746fd546f4ab74cc32bc72af6c: Log is configured to *not* fsync() on all Append() calls
I20260620 23:12:12.635977 20397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2555c2746fd546f4ab74cc32bc72af6c: No bootstrap required, opened a new log
I20260620 23:12:12.643172 20397 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2555c2746fd546f4ab74cc32bc72af6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2555c2746fd546f4ab74cc32bc72af6c" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39169 } }
I20260620 23:12:12.643571 20397 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2555c2746fd546f4ab74cc32bc72af6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:12:12.643663 20397 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2555c2746fd546f4ab74cc32bc72af6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2555c2746fd546f4ab74cc32bc72af6c, State: Initialized, Role: FOLLOWER
I20260620 23:12:12.644150 20397 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2555c2746fd546f4ab74cc32bc72af6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2555c2746fd546f4ab74cc32bc72af6c" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39169 } }
I20260620 23:12:12.644338 20397 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2555c2746fd546f4ab74cc32bc72af6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:12:12.644548 20397 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2555c2746fd546f4ab74cc32bc72af6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:12:12.644685 20397 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2555c2746fd546f4ab74cc32bc72af6c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:12:12.646697 20397 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2555c2746fd546f4ab74cc32bc72af6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2555c2746fd546f4ab74cc32bc72af6c" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39169 } }
I20260620 23:12:12.647186 20397 leader_election.cc:302] T 00000000000000000000000000000000 P 2555c2746fd546f4ab74cc32bc72af6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2555c2746fd546f4ab74cc32bc72af6c; no voters:
I20260620 23:12:12.647748 20397 leader_election.cc:288] T 00000000000000000000000000000000 P 2555c2746fd546f4ab74cc32bc72af6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:12:12.647908 20402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2555c2746fd546f4ab74cc32bc72af6c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:12:12.648607 20402 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2555c2746fd546f4ab74cc32bc72af6c [term 1 LEADER]: Becoming Leader. State: Replica: 2555c2746fd546f4ab74cc32bc72af6c, State: Running, Role: LEADER
I20260620 23:12:12.649147 20402 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2555c2746fd546f4ab74cc32bc72af6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2555c2746fd546f4ab74cc32bc72af6c" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39169 } }
I20260620 23:12:12.649832 20397 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2555c2746fd546f4ab74cc32bc72af6c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:12:12.651944 20404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2555c2746fd546f4ab74cc32bc72af6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2555c2746fd546f4ab74cc32bc72af6c. Latest consensus state: current_term: 1 leader_uuid: "2555c2746fd546f4ab74cc32bc72af6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2555c2746fd546f4ab74cc32bc72af6c" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39169 } } }
I20260620 23:12:12.652067 20403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2555c2746fd546f4ab74cc32bc72af6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2555c2746fd546f4ab74cc32bc72af6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2555c2746fd546f4ab74cc32bc72af6c" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39169 } } }
I20260620 23:12:12.652349 20404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2555c2746fd546f4ab74cc32bc72af6c [sys.catalog]: This master's current role is: LEADER
I20260620 23:12:12.652420 20403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2555c2746fd546f4ab74cc32bc72af6c [sys.catalog]: This master's current role is: LEADER
I20260620 23:12:12.653208 20410 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:12:12.658514 20410 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:12:12.667605 20410 catalog_manager.cc:1347] Generated new cluster ID: 7f6a497b49234eab98570d957b4800da
I20260620 23:12:12.667717 20410 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:12:12.697041 20410 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:12:12.698813 20410 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:12:12.706794 20410 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2555c2746fd546f4ab74cc32bc72af6c: Generated new TSK 0
I20260620 23:12:12.707830 20410 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:12:12.712855 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.193:0
--local_ip_for_outbound_sockets=127.10.191.193
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:39169
--builtin_ntp_servers=127.10.191.212:35899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 23:12:13.033413 20421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:12:13.033740 20421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:12:13.033898 20421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:12:13.041065 20421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:12:13.041198 20421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:12:13.041374 20421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.193
I20260620 23:12:13.050559 20421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:35899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:39169
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.193
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:12:13.051950 20421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:12:13.053669 20421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:12:13.066705 20427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:13.066780 20426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:13.066779 20429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:12:13.067402 20421 server_base.cc:1034] running on GCE node
I20260620 23:12:13.068032 20421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:12:13.068918 20421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:12:13.070161 20421 hybrid_clock.cc:648] HybridClock initialized: now 1781997133070120 us; error 62 us; skew 500 ppm
I20260620 23:12:13.070477 20421 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:12:13.072746 20421 webserver.cc:460] Webserver started at http://127.10.191.193:36653/ using document root <none> and password file <none>
I20260620 23:12:13.073427 20421 fs_manager.cc:359] Metadata directory not provided
I20260620 23:12:13.073557 20421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:12:13.073876 20421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:12:13.076632 20421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/data/instance:
uuid: "c8089c52bba84fa786afd19afdb483da"
format_stamp: "Formatted at 2026-06-20 23:12:13 on dist-test-slave-jkbq"
server_key: "afcb77c3ee6cf75655549abf0992c37c"
server_key_iv: "0a4aad8627c1ca0986e0e254903139ca"
server_key_version: "encryptionkey@0"
I20260620 23:12:13.077425 20421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance:
uuid: "c8089c52bba84fa786afd19afdb483da"
format_stamp: "Formatted at 2026-06-20 23:12:13 on dist-test-slave-jkbq"
server_key: "afcb77c3ee6cf75655549abf0992c37c"
server_key_iv: "0a4aad8627c1ca0986e0e254903139ca"
server_key_version: "encryptionkey@0"
I20260620 23:12:13.082446 20421 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:12:13.085836 20435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:13.087167 20421 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:12:13.087342 20421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/wal
uuid: "c8089c52bba84fa786afd19afdb483da"
format_stamp: "Formatted at 2026-06-20 23:12:13 on dist-test-slave-jkbq"
server_key: "afcb77c3ee6cf75655549abf0992c37c"
server_key_iv: "0a4aad8627c1ca0986e0e254903139ca"
server_key_version: "encryptionkey@0"
I20260620 23:12:13.087550 20421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:12:13.110631 20421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:12:13.111552 20421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:12:13.111871 20421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:12:13.112910 20421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:12:13.114346 20421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:12:13.114471 20421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:13.114605 20421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:12:13.114699 20421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:13.147734 20421 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.193:34813
I20260620 23:12:13.147830 20547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.193:34813 every 8 connection(s)
I20260620 23:12:13.149045 20421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
I20260620 23:12:13.156819 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 20421
I20260620 23:12:13.157003 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance
I20260620 23:12:13.157785 11007 external_mini_cluster.cc:1428] Setting key 85e15de9c446dd7c7f7eb09523b8e956
I20260620 23:12:13.160619 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.194:0
--local_ip_for_outbound_sockets=127.10.191.194
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:39169
--builtin_ntp_servers=127.10.191.212:35899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 23:12:13.162693 20550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:13.151903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:38335 (local address 127.10.191.254:39169)
0620 23:12:13.152311 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:13.152316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:13.153526 (+ 1210us) server_negotiation.cc:407] Connection header received
0620 23:12:13.155286 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:13.155290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:13.155354 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:13.155450 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:13.156607 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:13.157812 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:13.159894 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:13.160273 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:13.160377 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:13.161966 (+ 1589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:13.161971 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:13.161974 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:12:13.161990 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:13.162027 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:12:13.162039 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:13.162110 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:13.162376 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 23:12:13.162447 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":216,"threads_started":1}
I20260620 23:12:13.164176 20551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:13.152513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39169 (local address 127.10.191.193:38335)
0620 23:12:13.153372 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:13.153407 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:13.155044 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:13.155653 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:13.155661 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:13.155812 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:13.156407 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:13.156423 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:13.158030 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:13.158035 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:13.159728 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:13.159743 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:13.160500 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:13.160508 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:13.160850 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:13.161610 (+ 760us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:12:13.161616 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:13.161668 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:12:13.161776 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:12:13.162142 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:12:13.162156 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:12:13.162166 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 23:12:13.162342 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 23:12:13.162782 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":249,"threads_started":1}
I20260620 23:12:13.165629 20548 heartbeater.cc:347] Connected to a master server at 127.10.191.254:39169
I20260620 23:12:13.165916 20548 heartbeater.cc:464] Registering TS with master...
I20260620 23:12:13.166620 20548 heartbeater.cc:511] Master 127.10.191.254:39169 requested a full tablet report, sending...
I20260620 23:12:13.168547 20362 ts_manager.cc:194] Registered new tserver with Master: c8089c52bba84fa786afd19afdb483da (127.10.191.193:34813)
I20260620 23:12:13.170715 20362 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.191.193:38335
W20260620 23:12:13.478549 20552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:12:13.478821 20552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:12:13.478969 20552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:12:13.486235 20552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:12:13.486331 20552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:12:13.486434 20552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.194
I20260620 23:12:13.495199 20552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:35899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:39169
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.194
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:12:13.496436 20552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:12:13.498135 20552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:12:13.511044 20557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:13.511044 20558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:13.511198 20560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:12:13.511750 20552 server_base.cc:1034] running on GCE node
I20260620 23:12:13.512329 20552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:12:13.513289 20552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:12:13.514482 20552 hybrid_clock.cc:648] HybridClock initialized: now 1781997133514441 us; error 39 us; skew 500 ppm
I20260620 23:12:13.514788 20552 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:12:13.516986 20552 webserver.cc:460] Webserver started at http://127.10.191.194:44029/ using document root <none> and password file <none>
I20260620 23:12:13.517663 20552 fs_manager.cc:359] Metadata directory not provided
I20260620 23:12:13.517781 20552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:12:13.518100 20552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:12:13.520884 20552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/data/instance:
uuid: "642aeed9dfff43ceb647b9555163a40b"
format_stamp: "Formatted at 2026-06-20 23:12:13 on dist-test-slave-jkbq"
server_key: "506902bb59a9e279ae684fec6320d9a2"
server_key_iv: "9a26dfbaa4206189568af7f35bfda73d"
server_key_version: "encryptionkey@0"
I20260620 23:12:13.521667 20552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance:
uuid: "642aeed9dfff43ceb647b9555163a40b"
format_stamp: "Formatted at 2026-06-20 23:12:13 on dist-test-slave-jkbq"
server_key: "506902bb59a9e279ae684fec6320d9a2"
server_key_iv: "9a26dfbaa4206189568af7f35bfda73d"
server_key_version: "encryptionkey@0"
I20260620 23:12:13.526620 20552 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 23:12:13.529919 20566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:13.531281 20552 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260620 23:12:13.531451 20552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/wal
uuid: "642aeed9dfff43ceb647b9555163a40b"
format_stamp: "Formatted at 2026-06-20 23:12:13 on dist-test-slave-jkbq"
server_key: "506902bb59a9e279ae684fec6320d9a2"
server_key_iv: "9a26dfbaa4206189568af7f35bfda73d"
server_key_version: "encryptionkey@0"
I20260620 23:12:13.531664 20552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:12:13.557803 20552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:12:13.558669 20552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:12:13.559001 20552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:12:13.559930 20552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:12:13.561384 20552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:12:13.561506 20552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:13.561637 20552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:12:13.561728 20552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:13.594911 20552 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.194:37007
I20260620 23:12:13.594980 20678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.194:37007 every 8 connection(s)
I20260620 23:12:13.596148 20552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
I20260620 23:12:13.605154 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 20552
I20260620 23:12:13.605324 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance
I20260620 23:12:13.605840 11007 external_mini_cluster.cc:1428] Setting key 7a4328917383c853844265c6490af388
I20260620 23:12:13.608744 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.195:0
--local_ip_for_outbound_sockets=127.10.191.195
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:39169
--builtin_ntp_servers=127.10.191.212:35899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 23:12:13.609711 20681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:13.599139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39169 (local address 127.10.191.194:43827)
0620 23:12:13.600027 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:13.600063 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:13.601589 (+ 1526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:13.602165 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:13.602172 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:13.602263 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:13.602859 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:13.602939 (+ 80us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:13.603886 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:13.603890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:13.605420 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:13.605436 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:13.606193 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:13.606201 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:13.606494 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:13.607316 (+ 822us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:12:13.607318 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:13.607356 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:12:13.607431 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:12:13.607820 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:12:13.607829 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:12:13.607835 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 23:12:13.608014 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 23:12:13.608413 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":254,"threads_started":1}
I20260620 23:12:13.610579 20550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:13.598597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:43827 (local address 127.10.191.254:39169)
0620 23:12:13.598777 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:13.598782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:13.600174 (+ 1392us) server_negotiation.cc:407] Connection header received
0620 23:12:13.601824 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:13.601828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:13.601892 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:13.601975 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:13.603069 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:13.603754 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:13.605554 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:13.605960 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:13.606072 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:13.607634 (+ 1562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:13.607639 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:13.607642 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:12:13.607658 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:13.607687 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:12:13.607698 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:13.607809 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:13.608074 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 23:12:13.608219 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:12:13.611857 20679 heartbeater.cc:347] Connected to a master server at 127.10.191.254:39169
I20260620 23:12:13.612166 20679 heartbeater.cc:464] Registering TS with master...
I20260620 23:12:13.613078 20679 heartbeater.cc:511] Master 127.10.191.254:39169 requested a full tablet report, sending...
I20260620 23:12:13.614681 20362 ts_manager.cc:194] Registered new tserver with Master: 642aeed9dfff43ceb647b9555163a40b (127.10.191.194:37007)
I20260620 23:12:13.615790 20362 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.191.194:43827
W20260620 23:12:13.937202 20682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:12:13.937532 20682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:12:13.937693 20682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:12:13.945785 20682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:12:13.945888 20682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:12:13.946085 20682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.195
I20260620 23:12:13.956051 20682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:35899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:39169
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.195
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:12:13.957336 20682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:12:13.959189 20682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:12:13.972113 20690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:13.972122 20688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:13.972230 20687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:12:13.972966 20682 server_base.cc:1034] running on GCE node
I20260620 23:12:13.973542 20682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:12:13.974431 20682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:12:13.975647 20682 hybrid_clock.cc:648] HybridClock initialized: now 1781997133975614 us; error 74 us; skew 500 ppm
I20260620 23:12:13.975956 20682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:12:13.978235 20682 webserver.cc:460] Webserver started at http://127.10.191.195:37859/ using document root <none> and password file <none>
I20260620 23:12:13.979018 20682 fs_manager.cc:359] Metadata directory not provided
I20260620 23:12:13.979147 20682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:12:13.979465 20682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:12:13.982295 20682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/data/instance:
uuid: "31edf3eaf9b249cabb8c97d876d768d3"
format_stamp: "Formatted at 2026-06-20 23:12:13 on dist-test-slave-jkbq"
server_key: "3ea1d0ca76372b88c35fd66c6ea3bbba"
server_key_iv: "38dbf1fa4fed82941d1e5dd85592b4c0"
server_key_version: "encryptionkey@0"
I20260620 23:12:13.983177 20682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance:
uuid: "31edf3eaf9b249cabb8c97d876d768d3"
format_stamp: "Formatted at 2026-06-20 23:12:13 on dist-test-slave-jkbq"
server_key: "3ea1d0ca76372b88c35fd66c6ea3bbba"
server_key_iv: "38dbf1fa4fed82941d1e5dd85592b4c0"
server_key_version: "encryptionkey@0"
I20260620 23:12:13.988045 20682 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 23:12:13.991397 20696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:13.992772 20682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:12:13.992938 20682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/wal
uuid: "31edf3eaf9b249cabb8c97d876d768d3"
format_stamp: "Formatted at 2026-06-20 23:12:13 on dist-test-slave-jkbq"
server_key: "3ea1d0ca76372b88c35fd66c6ea3bbba"
server_key_iv: "38dbf1fa4fed82941d1e5dd85592b4c0"
server_key_version: "encryptionkey@0"
I20260620 23:12:13.993144 20682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:12:14.013213 20682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:12:14.014055 20682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:12:14.014358 20682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:12:14.015367 20682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:12:14.016798 20682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:12:14.016920 20682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:14.017050 20682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:12:14.017138 20682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:14.050365 20682 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.195:34107
I20260620 23:12:14.050453 20808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.195:34107 every 8 connection(s)
I20260620 23:12:14.051796 20682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
I20260620 23:12:14.057224 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 20682
I20260620 23:12:14.057385 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance
I20260620 23:12:14.057834 11007 external_mini_cluster.cc:1428] Setting key 148bfae05c1d01a2e975fc4644899190
I20260620 23:12:14.064323 20550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:14.054303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:53345 (local address 127.10.191.254:39169)
0620 23:12:14.054520 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:14.054526 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:14.056175 (+ 1649us) server_negotiation.cc:407] Connection header received
0620 23:12:14.058061 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:14.058068 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:14.058158 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:14.058272 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:14.059453 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:14.060185 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:14.061696 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:14.062096 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:14.062241 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:14.063523 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:14.063529 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:14.063534 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:12:14.063559 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:14.063587 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:12:14.063598 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:14.063692 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:14.063920 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 23:12:14.064007 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 23:12:14.065238 20811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:14.054905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39169 (local address 127.10.191.195:53345)
0620 23:12:14.055984 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:14.056036 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:14.057768 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:14.058504 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:14.058512 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:14.058598 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:14.059295 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:14.059311 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:14.060342 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:14.060348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:14.061551 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:14.061561 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:14.062316 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:14.062323 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:14.062564 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:14.063225 (+ 661us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:12:14.063227 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:14.063265 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:12:14.063341 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:12:14.063691 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:12:14.063704 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:12:14.063731 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 23:12:14.063896 (+ 165us) client_negotiation.cc:240] Negotiation successful
0620 23:12:14.064268 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":336,"threads_started":1}
I20260620 23:12:14.066493 20809 heartbeater.cc:347] Connected to a master server at 127.10.191.254:39169
I20260620 23:12:14.066812 20809 heartbeater.cc:464] Registering TS with master...
I20260620 23:12:14.067590 20809 heartbeater.cc:511] Master 127.10.191.254:39169 requested a full tablet report, sending...
I20260620 23:12:14.068884 20362 ts_manager.cc:194] Registered new tserver with Master: 31edf3eaf9b249cabb8c97d876d768d3 (127.10.191.195:34107)
I20260620 23:12:14.069669 20362 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.191.195:53345
I20260620 23:12:14.072760 11007 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 23:12:14.078573 20550 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:12:14.077092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33766 (local address 127.10.191.254:39169)
0620 23:12:14.077254 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:14.077258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:14.077733 (+ 475us) server_negotiation.cc:407] Connection header received
0620 23:12:14.077882 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:14.077886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:14.077956 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:14.078037 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:14.078406 (+ 369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33766: BlockingRecv error: recv got EOF from 127.0.0.1:33766 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 23:12:14.079234 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 20421
I20260620 23:12:14.087044 20818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:14.080215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40122 (local address 127.10.191.193:34813)
0620 23:12:14.080716 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:14.080724 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:14.080760 (+ 36us) server_negotiation.cc:407] Connection header received
0620 23:12:14.080819 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:14.080825 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:14.081046 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:14.081263 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:14.081614 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:14.082585 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:14.083907 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:14.084453 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:14.084588 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:14.085562 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:14.085570 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:14.085583 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:12:14.086154 (+ 571us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:14.086198 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:12:14.086227 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:14.086353 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:14.086491 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 23:12:14.086633 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":219,"threads_started":1}
I20260620 23:12:14.088310 20543 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:12:14.170277 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 20421
I20260620 23:12:14.186618 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 20552
I20260620 23:12:14.194149 20820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:14.187503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37884 (local address 127.10.191.194:37007)
0620 23:12:14.188023 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:14.188030 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:14.188057 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:12:14.188117 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:14.188124 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:14.188357 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:14.188485 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:14.188837 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:14.189797 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:14.191063 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:14.191611 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:14.191707 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:14.192727 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:14.192733 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:14.192746 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:12:14.193351 (+ 605us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:14.193394 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:12:14.193413 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:14.193498 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:14.193692 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 23:12:14.193820 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":264,"threads_started":1}
I20260620 23:12:14.195449 20674 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:12:14.276866 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 20552
I20260620 23:12:14.294710 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 20682
I20260620 23:12:14.303213 20822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:14.295771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53970 (local address 127.10.191.195:34107)
0620 23:12:14.296423 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:14.296431 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:14.296491 (+ 60us) server_negotiation.cc:407] Connection header received
0620 23:12:14.296552 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:14.296559 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:14.296788 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:14.296972 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:14.297441 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:14.298537 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:14.299907 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:14.300480 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:14.300616 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:14.301736 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:14.301744 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:14.301757 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:12:14.302334 (+ 577us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:14.302380 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:12:14.302399 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:14.302534 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:14.302704 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 23:12:14.302828 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":371,"threads_started":1}
I20260620 23:12:14.304510 20804 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:12:14.388099 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 20682
I20260620 23:12:14.403757 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 20331
I20260620 23:12:14.404930 20392 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:12:14.506397 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 20331
2026-06-20T23:12:14Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2371 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T23:12:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:12:14Z Disabled control of system clock
I20260620 23:12:14.577587 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:33347
--webserver_interface=127.10.191.254
--webserver_port=0
--builtin_ntp_servers=127.10.191.212:36381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:33347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 23:12:14.897560 20828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:12:14.897845 20828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:12:14.897929 20828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:12:14.905207 20828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:12:14.905311 20828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:12:14.905349 20828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:12:14.905411 20828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:12:14.905447 20828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:12:14.914467 20828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:36381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:33347
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:33347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:12:14.915823 20828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:12:14.917603 20828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:12:14.929303 20834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:14.929394 20836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:14.929682 20833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:12:14.931170 20828 server_base.cc:1034] running on GCE node
I20260620 23:12:14.931931 20828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:12:14.933316 20828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:12:14.934561 20828 hybrid_clock.cc:648] HybridClock initialized: now 1781997134934518 us; error 57 us; skew 500 ppm
I20260620 23:12:14.935055 20828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:12:14.937309 20828 webserver.cc:460] Webserver started at http://127.10.191.254:38169/ using document root <none> and password file <none>
I20260620 23:12:14.938016 20828 fs_manager.cc:359] Metadata directory not provided
I20260620 23:12:14.938144 20828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:12:14.938491 20828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:12:14.941540 20828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "404542b92d5044c0a21c3b1f59bf11b2"
format_stamp: "Formatted at 2026-06-20 23:12:14 on dist-test-slave-jkbq"
server_key: "53335de3674545e0ae39304d71a54b9e"
server_key_iv: "46b49ef97c004f6abd0493abe0e93e9b"
server_key_version: "encryptionkey@0"
I20260620 23:12:14.942375 20828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "404542b92d5044c0a21c3b1f59bf11b2"
format_stamp: "Formatted at 2026-06-20 23:12:14 on dist-test-slave-jkbq"
server_key: "53335de3674545e0ae39304d71a54b9e"
server_key_iv: "46b49ef97c004f6abd0493abe0e93e9b"
server_key_version: "encryptionkey@0"
I20260620 23:12:14.947678 20828 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 23:12:14.951220 20842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:14.952549 20828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:12:14.952744 20828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "404542b92d5044c0a21c3b1f59bf11b2"
format_stamp: "Formatted at 2026-06-20 23:12:14 on dist-test-slave-jkbq"
server_key: "53335de3674545e0ae39304d71a54b9e"
server_key_iv: "46b49ef97c004f6abd0493abe0e93e9b"
server_key_version: "encryptionkey@0"
I20260620 23:12:14.952975 20828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:12:14.958607 20828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:12:14.959431 20828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:12:14.959724 20828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:12:14.977738 20828 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:33347
I20260620 23:12:14.977782 20893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:33347 every 8 connection(s)
I20260620 23:12:14.979111 20828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:12:14.982674 20894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:12:14.986220 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 20828
I20260620 23:12:14.986429 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:12:14.987322 11007 external_mini_cluster.cc:1428] Setting key 791977c94d6f6fca84131a675b8f61b4
I20260620 23:12:14.995596 20894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 404542b92d5044c0a21c3b1f59bf11b2: Bootstrap starting.
I20260620 23:12:14.995643 20897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:14.989388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48000 (local address 127.10.191.254:33347)
0620 23:12:14.990441 (+ 1053us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:14.990452 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:14.990494 (+ 42us) server_negotiation.cc:407] Connection header received
0620 23:12:14.992069 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:14.992091 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:14.992525 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:14.992945 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:14.994015 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:14.994020 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:14.994032 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:12:14.994045 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:14.994098 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:12:14.994112 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:14.994349 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:14.994478 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 23:12:14.994710 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":408,"threads_started":1}
I20260620 23:12:14.998637 20894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 404542b92d5044c0a21c3b1f59bf11b2: Neither blocks nor log segments found. Creating new log.
I20260620 23:12:14.999634 20894 log.cc:824] T 00000000000000000000000000000000 P 404542b92d5044c0a21c3b1f59bf11b2: Log is configured to *not* fsync() on all Append() calls
I20260620 23:12:15.002247 20894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 404542b92d5044c0a21c3b1f59bf11b2: No bootstrap required, opened a new log
I20260620 23:12:15.009014 20894 raft_consensus.cc:348] T 00000000000000000000000000000000 P 404542b92d5044c0a21c3b1f59bf11b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "404542b92d5044c0a21c3b1f59bf11b2" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 33347 } }
I20260620 23:12:15.009361 20894 raft_consensus.cc:374] T 00000000000000000000000000000000 P 404542b92d5044c0a21c3b1f59bf11b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:12:15.009465 20894 raft_consensus.cc:729] T 00000000000000000000000000000000 P 404542b92d5044c0a21c3b1f59bf11b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 404542b92d5044c0a21c3b1f59bf11b2, State: Initialized, Role: FOLLOWER
I20260620 23:12:15.009979 20894 consensus_queue.cc:260] T 00000000000000000000000000000000 P 404542b92d5044c0a21c3b1f59bf11b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "404542b92d5044c0a21c3b1f59bf11b2" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 33347 } }
I20260620 23:12:15.010202 20894 raft_consensus.cc:388] T 00000000000000000000000000000000 P 404542b92d5044c0a21c3b1f59bf11b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:12:15.010341 20894 raft_consensus.cc:482] T 00000000000000000000000000000000 P 404542b92d5044c0a21c3b1f59bf11b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:12:15.010501 20894 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 404542b92d5044c0a21c3b1f59bf11b2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:12:15.012545 20894 raft_consensus.cc:504] T 00000000000000000000000000000000 P 404542b92d5044c0a21c3b1f59bf11b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "404542b92d5044c0a21c3b1f59bf11b2" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 33347 } }
I20260620 23:12:15.013031 20894 leader_election.cc:302] T 00000000000000000000000000000000 P 404542b92d5044c0a21c3b1f59bf11b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 404542b92d5044c0a21c3b1f59bf11b2; no voters:
I20260620 23:12:15.013604 20894 leader_election.cc:288] T 00000000000000000000000000000000 P 404542b92d5044c0a21c3b1f59bf11b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:12:15.013707 20899 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 404542b92d5044c0a21c3b1f59bf11b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:12:15.014500 20899 raft_consensus.cc:686] T 00000000000000000000000000000000 P 404542b92d5044c0a21c3b1f59bf11b2 [term 1 LEADER]: Becoming Leader. State: Replica: 404542b92d5044c0a21c3b1f59bf11b2, State: Running, Role: LEADER
I20260620 23:12:15.015082 20899 consensus_queue.cc:237] T 00000000000000000000000000000000 P 404542b92d5044c0a21c3b1f59bf11b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "404542b92d5044c0a21c3b1f59bf11b2" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 33347 } }
I20260620 23:12:15.015805 20894 sys_catalog.cc:564] T 00000000000000000000000000000000 P 404542b92d5044c0a21c3b1f59bf11b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:12:15.017779 20901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 404542b92d5044c0a21c3b1f59bf11b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 404542b92d5044c0a21c3b1f59bf11b2. Latest consensus state: current_term: 1 leader_uuid: "404542b92d5044c0a21c3b1f59bf11b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "404542b92d5044c0a21c3b1f59bf11b2" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 33347 } } }
I20260620 23:12:15.018081 20901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 404542b92d5044c0a21c3b1f59bf11b2 [sys.catalog]: This master's current role is: LEADER
I20260620 23:12:15.017938 20900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 404542b92d5044c0a21c3b1f59bf11b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "404542b92d5044c0a21c3b1f59bf11b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "404542b92d5044c0a21c3b1f59bf11b2" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 33347 } } }
I20260620 23:12:15.018353 20900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 404542b92d5044c0a21c3b1f59bf11b2 [sys.catalog]: This master's current role is: LEADER
I20260620 23:12:15.022141 20904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:12:15.027828 20904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:12:15.036983 20904 catalog_manager.cc:1347] Generated new cluster ID: 721e76886d354e089e8dad016ebcf390
I20260620 23:12:15.037109 20904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:12:15.052347 20904 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:12:15.052758 20904 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:12:15.061197 20904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 404542b92d5044c0a21c3b1f59bf11b2: Generated new TSK 0
I20260620 23:12:15.062111 20904 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:12:15.076907 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.193:0
--local_ip_for_outbound_sockets=127.10.191.193
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:33347
--builtin_ntp_servers=127.10.191.212:36381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 23:12:15.418746 20918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:12:15.419086 20918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:12:15.419286 20918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:12:15.426687 20918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:12:15.426789 20918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:12:15.426981 20918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.193
I20260620 23:12:15.436190 20918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:36381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:33347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.193
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:12:15.437924 20918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:12:15.439767 20918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:12:15.453148 20923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:15.453219 20924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:15.453117 20926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:12:15.456346 20918 server_base.cc:1034] running on GCE node
I20260620 23:12:15.456945 20918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:12:15.457942 20918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:12:15.459167 20918 hybrid_clock.cc:648] HybridClock initialized: now 1781997135459129 us; error 58 us; skew 500 ppm
I20260620 23:12:15.459481 20918 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:12:15.462159 20918 webserver.cc:460] Webserver started at http://127.10.191.193:37637/ using document root <none> and password file <none>
I20260620 23:12:15.462963 20918 fs_manager.cc:359] Metadata directory not provided
I20260620 23:12:15.463074 20918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:12:15.463385 20918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:12:15.466137 20918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/data/instance:
uuid: "533d705fccce43cd93da598beff54bf9"
format_stamp: "Formatted at 2026-06-20 23:12:15 on dist-test-slave-jkbq"
server_key: "830b15cea2f11a9c6bbfe0b30aa900af"
server_key_iv: "96b9b4973726fc7e7fa6d4f7223916c6"
server_key_version: "encryptionkey@0"
I20260620 23:12:15.466987 20918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance:
uuid: "533d705fccce43cd93da598beff54bf9"
format_stamp: "Formatted at 2026-06-20 23:12:15 on dist-test-slave-jkbq"
server_key: "830b15cea2f11a9c6bbfe0b30aa900af"
server_key_iv: "96b9b4973726fc7e7fa6d4f7223916c6"
server_key_version: "encryptionkey@0"
I20260620 23:12:15.471959 20918 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 23:12:15.475229 20932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:15.476480 20918 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 23:12:15.476632 20918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/wal
uuid: "533d705fccce43cd93da598beff54bf9"
format_stamp: "Formatted at 2026-06-20 23:12:15 on dist-test-slave-jkbq"
server_key: "830b15cea2f11a9c6bbfe0b30aa900af"
server_key_iv: "96b9b4973726fc7e7fa6d4f7223916c6"
server_key_version: "encryptionkey@0"
I20260620 23:12:15.476864 20918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:12:15.482187 20918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:12:15.483040 20918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:12:15.483301 20918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:12:15.484158 20918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:12:15.485445 20918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:12:15.485554 20918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:15.485678 20918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:12:15.485783 20918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:15.518488 20918 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.193:41209
I20260620 23:12:15.518564 21044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.193:41209 every 8 connection(s)
I20260620 23:12:15.520097 20918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
I20260620 23:12:15.522127 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 20918
I20260620 23:12:15.522341 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance
I20260620 23:12:15.523219 11007 external_mini_cluster.cc:1428] Setting key a9213fe488db30b64195ca9920832a85
I20260620 23:12:15.525858 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.194:0
--local_ip_for_outbound_sockets=127.10.191.194
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:33347
--builtin_ntp_servers=127.10.191.212:36381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 23:12:15.529072 21047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:15.522775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:55081 (local address 127.10.191.254:33347)
0620 23:12:15.523285 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:15.523291 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:15.524788 (+ 1497us) server_negotiation.cc:407] Connection header received
0620 23:12:15.526582 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:15.526591 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:15.526656 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:15.526786 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:15.528187 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:15.528195 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:15.528202 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:12:15.528224 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:15.528276 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:12:15.528293 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:15.528434 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:15.528648 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 23:12:15.528720 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":256,"threads_started":1}
I20260620 23:12:15.529870 21048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:15.523512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:33347 (local address 127.10.191.193:55081)
0620 23:12:15.524618 (+ 1106us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:15.524663 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:15.526319 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:15.527058 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:15.527066 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:15.527161 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:15.527892 (+ 731us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:12:15.527895 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:15.527932 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:12:15.528014 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:12:15.528387 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:12:15.528393 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:12:15.528399 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 23:12:15.528721 (+ 322us) client_negotiation.cc:240] Negotiation successful
0620 23:12:15.528924 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":341,"threads_started":1}
I20260620 23:12:15.531363 21045 heartbeater.cc:347] Connected to a master server at 127.10.191.254:33347
I20260620 23:12:15.531677 21045 heartbeater.cc:464] Registering TS with master...
I20260620 23:12:15.532374 21045 heartbeater.cc:511] Master 127.10.191.254:33347 requested a full tablet report, sending...
I20260620 23:12:15.534680 20859 ts_manager.cc:194] Registered new tserver with Master: 533d705fccce43cd93da598beff54bf9 (127.10.191.193:41209)
W20260620 23:12:15.848374 21049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:12:15.848654 21049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:12:15.848747 21049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:12:15.856098 21049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:12:15.856216 21049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:12:15.856326 21049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.194
I20260620 23:12:15.865231 21049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:36381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:33347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.194
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:12:15.866441 21049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:12:15.868160 21049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:12:15.880760 21054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:15.880821 21055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:15.880872 21057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:12:15.881848 21049 server_base.cc:1034] running on GCE node
I20260620 23:12:15.882421 21049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:12:15.883419 21049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:12:15.884647 21049 hybrid_clock.cc:648] HybridClock initialized: now 1781997135884616 us; error 48 us; skew 500 ppm
I20260620 23:12:15.884963 21049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:12:15.887169 21049 webserver.cc:460] Webserver started at http://127.10.191.194:33563/ using document root <none> and password file <none>
I20260620 23:12:15.887848 21049 fs_manager.cc:359] Metadata directory not provided
I20260620 23:12:15.888190 21049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:12:15.888571 21049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:12:15.891379 21049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/data/instance:
uuid: "90fbbceb575042c7bca9f33b4b65ab9a"
format_stamp: "Formatted at 2026-06-20 23:12:15 on dist-test-slave-jkbq"
server_key: "bcf33e3338c7a067abc7da2a30dfa70c"
server_key_iv: "34ac84293f8afd3a52b6f3b2cdaeefdb"
server_key_version: "encryptionkey@0"
I20260620 23:12:15.892164 21049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance:
uuid: "90fbbceb575042c7bca9f33b4b65ab9a"
format_stamp: "Formatted at 2026-06-20 23:12:15 on dist-test-slave-jkbq"
server_key: "bcf33e3338c7a067abc7da2a30dfa70c"
server_key_iv: "34ac84293f8afd3a52b6f3b2cdaeefdb"
server_key_version: "encryptionkey@0"
I20260620 23:12:15.897121 21049 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:12:15.900511 21063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:15.901767 21049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:12:15.901942 21049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/wal
uuid: "90fbbceb575042c7bca9f33b4b65ab9a"
format_stamp: "Formatted at 2026-06-20 23:12:15 on dist-test-slave-jkbq"
server_key: "bcf33e3338c7a067abc7da2a30dfa70c"
server_key_iv: "34ac84293f8afd3a52b6f3b2cdaeefdb"
server_key_version: "encryptionkey@0"
I20260620 23:12:15.902160 21049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:12:15.908350 21049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:12:15.909102 21049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:12:15.909368 21049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:12:15.910238 21049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:12:15.911625 21049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:12:15.911756 21049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:15.911891 21049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:12:15.911991 21049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:15.944763 21049 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.194:37173
I20260620 23:12:15.944815 21175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.194:37173 every 8 connection(s)
I20260620 23:12:15.946070 21049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
I20260620 23:12:15.949676 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 21049
I20260620 23:12:15.949898 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance
I20260620 23:12:15.950352 11007 external_mini_cluster.cc:1428] Setting key 96d9141912ed8a4d81edf0001af58d26
I20260620 23:12:15.953230 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.195:0
--local_ip_for_outbound_sockets=127.10.191.195
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:33347
--builtin_ntp_servers=127.10.191.212:36381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 23:12:15.954952 21047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:15.948838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:53823 (local address 127.10.191.254:33347)
0620 23:12:15.949062 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:15.949069 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:15.950618 (+ 1549us) server_negotiation.cc:407] Connection header received
0620 23:12:15.952586 (+ 1968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:15.952593 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:15.952653 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:15.952765 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:15.954060 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:15.954069 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:15.954077 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:12:15.954101 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:15.954145 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:12:15.954163 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:15.954299 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:15.954481 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 23:12:15.954549 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 23:12:15.955742 21178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:15.949382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:33347 (local address 127.10.191.194:53823)
0620 23:12:15.950436 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:15.950486 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:15.952332 (+ 1846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:15.952954 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:15.952962 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:15.953055 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:15.953759 (+ 704us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:12:15.953761 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:15.953799 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:12:15.953884 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:12:15.954276 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:12:15.954282 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:12:15.954288 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 23:12:15.954491 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 23:12:15.954684 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":355,"threads_started":1}
I20260620 23:12:15.957180 21176 heartbeater.cc:347] Connected to a master server at 127.10.191.254:33347
I20260620 23:12:15.957518 21176 heartbeater.cc:464] Registering TS with master...
I20260620 23:12:15.958209 21176 heartbeater.cc:511] Master 127.10.191.254:33347 requested a full tablet report, sending...
I20260620 23:12:15.959735 20859 ts_manager.cc:194] Registered new tserver with Master: 90fbbceb575042c7bca9f33b4b65ab9a (127.10.191.194:37173)
W20260620 23:12:16.279449 21179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:12:16.279702 21179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:12:16.279824 21179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:12:16.289593 21179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:12:16.289693 21179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:12:16.289801 21179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.195
I20260620 23:12:16.298640 21179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:36381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:33347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.195
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:12:16.299934 21179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:12:16.301627 21179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:12:16.314400 21184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:16.314376 21185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:16.314337 21187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:12:16.315614 21179 server_base.cc:1034] running on GCE node
I20260620 23:12:16.316254 21179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:12:16.317433 21179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:12:16.318660 21179 hybrid_clock.cc:648] HybridClock initialized: now 1781997136318590 us; error 52 us; skew 500 ppm
I20260620 23:12:16.319096 21179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:12:16.321573 21179 webserver.cc:460] Webserver started at http://127.10.191.195:38487/ using document root <none> and password file <none>
I20260620 23:12:16.322585 21179 fs_manager.cc:359] Metadata directory not provided
I20260620 23:12:16.322759 21179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:12:16.323200 21179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:12:16.327147 21179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/data/instance:
uuid: "b96d2441eb874410be2969a99c0672a1"
format_stamp: "Formatted at 2026-06-20 23:12:16 on dist-test-slave-jkbq"
server_key: "123d3fe325daf14408a5d3752595c30c"
server_key_iv: "80cae90373f6ae8aad1da64434325317"
server_key_version: "encryptionkey@0"
I20260620 23:12:16.328003 21179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance:
uuid: "b96d2441eb874410be2969a99c0672a1"
format_stamp: "Formatted at 2026-06-20 23:12:16 on dist-test-slave-jkbq"
server_key: "123d3fe325daf14408a5d3752595c30c"
server_key_iv: "80cae90373f6ae8aad1da64434325317"
server_key_version: "encryptionkey@0"
I20260620 23:12:16.333214 21179 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:12:16.336699 21193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:16.338029 21179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:12:16.338201 21179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/wal
uuid: "b96d2441eb874410be2969a99c0672a1"
format_stamp: "Formatted at 2026-06-20 23:12:16 on dist-test-slave-jkbq"
server_key: "123d3fe325daf14408a5d3752595c30c"
server_key_iv: "80cae90373f6ae8aad1da64434325317"
server_key_version: "encryptionkey@0"
I20260620 23:12:16.338433 21179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:12:16.344290 21179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:12:16.345116 21179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:12:16.345398 21179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:12:16.346310 21179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:12:16.347779 21179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:12:16.347896 21179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:16.348034 21179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:12:16.348132 21179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:16.380548 21179 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.195:46753
I20260620 23:12:16.380679 21305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.195:46753 every 8 connection(s)
I20260620 23:12:16.381953 21179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
I20260620 23:12:16.386029 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 21179
I20260620 23:12:16.386210 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance
I20260620 23:12:16.386643 11007 external_mini_cluster.cc:1428] Setting key 381715c90ff0db6e228ff95f0fbfe926
I20260620 23:12:16.390372 21047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:16.384511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:33015 (local address 127.10.191.254:33347)
0620 23:12:16.384684 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:16.384689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:16.386092 (+ 1403us) server_negotiation.cc:407] Connection header received
0620 23:12:16.388068 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:16.388073 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:16.388144 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:16.388262 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:16.389642 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:16.389648 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:16.389653 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:12:16.389677 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:16.389706 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:12:16.389717 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:16.389843 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:16.390043 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 23:12:16.390106 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 23:12:16.391382 21308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:16.384965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:33347 (local address 127.10.191.195:33015)
0620 23:12:16.385918 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:16.385953 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:16.387781 (+ 1828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:16.388497 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:16.388505 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:16.388626 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:16.389317 (+ 691us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:12:16.389320 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:16.389377 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:12:16.389483 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:12:16.389825 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:12:16.389835 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:12:16.389842 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 23:12:16.390129 (+ 287us) client_negotiation.cc:240] Negotiation successful
0620 23:12:16.390377 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":290,"threads_started":1}
I20260620 23:12:16.392530 21306 heartbeater.cc:347] Connected to a master server at 127.10.191.254:33347
I20260620 23:12:16.392828 21306 heartbeater.cc:464] Registering TS with master...
I20260620 23:12:16.393391 21306 heartbeater.cc:511] Master 127.10.191.254:33347 requested a full tablet report, sending...
I20260620 23:12:16.394670 20859 ts_manager.cc:194] Registered new tserver with Master: b96d2441eb874410be2969a99c0672a1 (127.10.191.195:46753)
I20260620 23:12:16.401829 11007 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 23:12:16.407850 21047 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:12:16.406261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48002 (local address 127.10.191.254:33347)
0620 23:12:16.406424 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:16.406428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:16.407075 (+ 647us) server_negotiation.cc:407] Connection header received
0620 23:12:16.407211 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:16.407215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:16.407263 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:16.407340 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:16.407602 (+ 262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48002: BlockingRecv error: recv got EOF from 127.0.0.1:48002 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 23:12:16.409202 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 20918
I20260620 23:12:16.412459 21315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:16.410042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56192 (local address 127.10.191.193:41209)
0620 23:12:16.410589 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:16.410597 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:16.410622 (+ 25us) server_negotiation.cc:407] Connection header received
0620 23:12:16.410675 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:16.410682 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:16.410953 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:16.411088 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:16.411885 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:16.411892 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:16.411903 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:12:16.411919 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:16.411959 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:12:16.411972 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:16.412072 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:16.412148 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 23:12:16.412222 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":285,"threads_started":1}
I20260620 23:12:16.413750 21040 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:12:16.494844 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 20918
I20260620 23:12:16.511142 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 21049
I20260620 23:12:16.514529 21317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:16.512033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52292 (local address 127.10.191.194:37173)
0620 23:12:16.512516 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:16.512523 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:16.512550 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:12:16.512612 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:16.512618 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:16.512828 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:16.512953 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:16.513818 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:16.513825 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:16.513836 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:12:16.513849 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:16.513889 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:12:16.513904 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:16.514006 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:16.514109 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 23:12:16.514241 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":236,"threads_started":1}
I20260620 23:12:16.515787 21171 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:12:16.595980 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 21049
I20260620 23:12:16.611593 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 21179
I20260620 23:12:16.615587 21319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:16.612522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38122 (local address 127.10.191.195:46753)
0620 23:12:16.613130 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:16.613137 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:16.613167 (+ 30us) server_negotiation.cc:407] Connection header received
0620 23:12:16.613228 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:16.613234 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:16.613457 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:16.613633 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:16.614779 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:16.614785 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:16.614796 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:12:16.614812 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:16.614850 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:12:16.614939 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:16.615127 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:16.615200 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 23:12:16.615297 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":325,"threads_started":1}
I20260620 23:12:16.616858 21301 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:12:16.701287 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 21179
I20260620 23:12:16.716666 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 20828
I20260620 23:12:16.717869 20889 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:12:16.817606 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 20828
2026-06-20T23:12:16Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2308 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:12:16 dist-test-slave-jkbq krb5kdc[21324](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:12:16 dist-test-slave-jkbq krb5kdc[21324](info): set up 2 sockets
Jun 20 23:12:16 dist-test-slave-jkbq krb5kdc[21324](info): commencing operation
krb5kdc: starting...
W20260620 23:12:17.923861 11007 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Jun 20 23:12:18 dist-test-slave-jkbq krb5kdc[21324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:12:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:12:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.191.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
I20260620 23:12:18.146039 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:33655
--webserver_interface=127.10.191.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:39441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:33655
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:12:18.464123 21337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:12:18.464383 21337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:12:18.464470 21337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:12:18.471558 21337 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:12:18.471658 21337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:12:18.471704 21337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:12:18.471761 21337 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:12:18.471807 21337 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:12:18.481248 21337 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:39441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:33655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:33655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:12:18.482419 21337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:12:18.483703 21337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:12:18.494758 21342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:18.494884 21343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:18.494884 21345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:12:18.495659 21337 server_base.cc:1034] running on GCE node
I20260620 23:12:18.496400 21337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:12:18.497730 21337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:12:18.498968 21337 hybrid_clock.cc:648] HybridClock initialized: now 1781997138498913 us; error 61 us; skew 500 ppm
I20260620 23:12:18.499295 21337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:12:18 dist-test-slave-jkbq krb5kdc[21324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997138, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:12:18.502187 21337 init.cc:377] Logged in from keytab as kudu/127.10.191.254@KRBTEST.COM (short username kudu)
I20260620 23:12:18.503700 21337 webserver.cc:460] Webserver started at http://127.10.191.254:38239/ using document root <none> and password file <none>
I20260620 23:12:18.504330 21337 fs_manager.cc:359] Metadata directory not provided
I20260620 23:12:18.504477 21337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:12:18.504784 21337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:12:18.507675 21337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "94a04d2d407a4a8aa0df53e94e3a4d3a"
format_stamp: "Formatted at 2026-06-20 23:12:18 on dist-test-slave-jkbq"
server_key: "aa559a808897aef6f66e99b72f71e04e"
server_key_iv: "7b4cc8d4b246eda690383ed975547454"
server_key_version: "encryptionkey@0"
I20260620 23:12:18.508391 21337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "94a04d2d407a4a8aa0df53e94e3a4d3a"
format_stamp: "Formatted at 2026-06-20 23:12:18 on dist-test-slave-jkbq"
server_key: "aa559a808897aef6f66e99b72f71e04e"
server_key_iv: "7b4cc8d4b246eda690383ed975547454"
server_key_version: "encryptionkey@0"
I20260620 23:12:18.513119 21337 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 23:12:18.516314 21352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:18.517517 21337 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 23:12:18.517704 21337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "94a04d2d407a4a8aa0df53e94e3a4d3a"
format_stamp: "Formatted at 2026-06-20 23:12:18 on dist-test-slave-jkbq"
server_key: "aa559a808897aef6f66e99b72f71e04e"
server_key_iv: "7b4cc8d4b246eda690383ed975547454"
server_key_version: "encryptionkey@0"
I20260620 23:12:18.517894 21337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:12:18.531956 21337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:12:18.535617 21337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:12:18.535892 21337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:12:18.553423 21337 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:33655
I20260620 23:12:18.553455 21403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:33655 every 8 connection(s)
I20260620 23:12:18.554702 21337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:12:18.557047 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 21337
I20260620 23:12:18.557281 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:12:18.557943 11007 external_mini_cluster.cc:1428] Setting key 807fb0aaa2bd84dcdc44b39d055bca64
I20260620 23:12:18.558324 21404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:12:18.571851 21404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94a04d2d407a4a8aa0df53e94e3a4d3a: Bootstrap starting.
Jun 20 23:12:18 dist-test-slave-jkbq krb5kdc[21324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:12:18.575412 21404 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94a04d2d407a4a8aa0df53e94e3a4d3a: Neither blocks nor log segments found. Creating new log.
I20260620 23:12:18.576678 21404 log.cc:824] T 00000000000000000000000000000000 P 94a04d2d407a4a8aa0df53e94e3a4d3a: Log is configured to *not* fsync() on all Append() calls
I20260620 23:12:18.579981 21404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94a04d2d407a4a8aa0df53e94e3a4d3a: No bootstrap required, opened a new log
I20260620 23:12:18.581511 21407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:18.559921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58554 (local address 127.10.191.254:33655)
0620 23:12:18.560725 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:18.560739 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:18.560786 (+ 47us) server_negotiation.cc:407] Connection header received
0620 23:12:18.563023 (+ 2237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:18.563051 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:18.563534 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:18.563944 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:18.565141 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:18.567100 (+ 1959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:18.568287 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:18.569350 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:18.569543 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:18.573344 (+ 3801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:18.573354 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:18.573372 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:18.573423 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:18.577172 (+ 3749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:18.578031 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:18.578039 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:18.578046 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:18.578223 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:18.578584 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:18.578589 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:18.578592 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:18.579300 (+ 708us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:18.579606 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:18.579876 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 23:12:18.580265 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":348,"threads_started":1}
I20260620 23:12:18.588313 21404 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94a04d2d407a4a8aa0df53e94e3a4d3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94a04d2d407a4a8aa0df53e94e3a4d3a" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 33655 } }
I20260620 23:12:18.588601 21404 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94a04d2d407a4a8aa0df53e94e3a4d3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:12:18.588750 21404 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94a04d2d407a4a8aa0df53e94e3a4d3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94a04d2d407a4a8aa0df53e94e3a4d3a, State: Initialized, Role: FOLLOWER
I20260620 23:12:18.589288 21404 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94a04d2d407a4a8aa0df53e94e3a4d3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94a04d2d407a4a8aa0df53e94e3a4d3a" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 33655 } }
I20260620 23:12:18.589504 21404 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94a04d2d407a4a8aa0df53e94e3a4d3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:12:18.589625 21404 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94a04d2d407a4a8aa0df53e94e3a4d3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:12:18.589793 21404 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94a04d2d407a4a8aa0df53e94e3a4d3a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:12:18.591914 21404 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94a04d2d407a4a8aa0df53e94e3a4d3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94a04d2d407a4a8aa0df53e94e3a4d3a" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 33655 } }
I20260620 23:12:18.592377 21404 leader_election.cc:302] T 00000000000000000000000000000000 P 94a04d2d407a4a8aa0df53e94e3a4d3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94a04d2d407a4a8aa0df53e94e3a4d3a; no voters:
I20260620 23:12:18.592901 21404 leader_election.cc:288] T 00000000000000000000000000000000 P 94a04d2d407a4a8aa0df53e94e3a4d3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:12:18.592993 21409 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94a04d2d407a4a8aa0df53e94e3a4d3a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:12:18.593850 21409 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94a04d2d407a4a8aa0df53e94e3a4d3a [term 1 LEADER]: Becoming Leader. State: Replica: 94a04d2d407a4a8aa0df53e94e3a4d3a, State: Running, Role: LEADER
I20260620 23:12:18.594554 21409 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94a04d2d407a4a8aa0df53e94e3a4d3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94a04d2d407a4a8aa0df53e94e3a4d3a" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 33655 } }
I20260620 23:12:18.594851 21404 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94a04d2d407a4a8aa0df53e94e3a4d3a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:12:18.597190 21410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94a04d2d407a4a8aa0df53e94e3a4d3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94a04d2d407a4a8aa0df53e94e3a4d3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94a04d2d407a4a8aa0df53e94e3a4d3a" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 33655 } } }
I20260620 23:12:18.597560 21410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94a04d2d407a4a8aa0df53e94e3a4d3a [sys.catalog]: This master's current role is: LEADER
I20260620 23:12:18.597345 21411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94a04d2d407a4a8aa0df53e94e3a4d3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94a04d2d407a4a8aa0df53e94e3a4d3a. Latest consensus state: current_term: 1 leader_uuid: "94a04d2d407a4a8aa0df53e94e3a4d3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94a04d2d407a4a8aa0df53e94e3a4d3a" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 33655 } } }
I20260620 23:12:18.597731 21411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94a04d2d407a4a8aa0df53e94e3a4d3a [sys.catalog]: This master's current role is: LEADER
I20260620 23:12:18.601400 21418 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:12:18.606109 21418 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:12:18.614338 21418 catalog_manager.cc:1347] Generated new cluster ID: fd5a1628ea3943c399f4a569db3627bf
I20260620 23:12:18.614436 21418 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:12:18.644383 21418 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:12:18.646179 21418 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:12:18.655282 21418 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94a04d2d407a4a8aa0df53e94e3a4d3a: Generated new TSK 0
I20260620 23:12:18.656086 21418 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.191.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab.
Entry for principal kudu/127.10.191.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab.
I20260620 23:12:18.722106 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.193:0
--local_ip_for_outbound_sockets=127.10.191.193
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:33655
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:39441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:12:19.064723 21430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:12:19.064986 21430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:12:19.065145 21430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:12:19.072366 21430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:12:19.072477 21430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:12:19.072665 21430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.193
I20260620 23:12:19.082320 21430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:39441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:33655
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.193
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:12:19.083825 21430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:12:19.085142 21430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:12:19.098636 21438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:19.098640 21436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:19.099340 21435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:12:19.099447 21430 server_base.cc:1034] running on GCE node
I20260620 23:12:19.100138 21430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:12:19.101018 21430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:12:19.102250 21430 hybrid_clock.cc:648] HybridClock initialized: now 1781997139102211 us; error 40 us; skew 500 ppm
I20260620 23:12:19.102586 21430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:12:19 dist-test-slave-jkbq krb5kdc[21324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997139, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:12:19.105772 21430 init.cc:377] Logged in from keytab as kudu/127.10.191.193@KRBTEST.COM (short username kudu)
I20260620 23:12:19.107271 21430 webserver.cc:460] Webserver started at http://127.10.191.193:39297/ using document root <none> and password file <none>
I20260620 23:12:19.107925 21430 fs_manager.cc:359] Metadata directory not provided
I20260620 23:12:19.108058 21430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:12:19.108367 21430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:12:19.111274 21430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/data/instance:
uuid: "ed18b2da9d8348008d41f632748f140c"
format_stamp: "Formatted at 2026-06-20 23:12:19 on dist-test-slave-jkbq"
server_key: "fbcdbb5a4fa74e020774ff606d6ce4cc"
server_key_iv: "be100d16176d2327559988da02678b5f"
server_key_version: "encryptionkey@0"
I20260620 23:12:19.112025 21430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance:
uuid: "ed18b2da9d8348008d41f632748f140c"
format_stamp: "Formatted at 2026-06-20 23:12:19 on dist-test-slave-jkbq"
server_key: "fbcdbb5a4fa74e020774ff606d6ce4cc"
server_key_iv: "be100d16176d2327559988da02678b5f"
server_key_version: "encryptionkey@0"
I20260620 23:12:19.117065 21430 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:12:19.120456 21445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:19.121956 21430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:12:19.122193 21430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/wal
uuid: "ed18b2da9d8348008d41f632748f140c"
format_stamp: "Formatted at 2026-06-20 23:12:19 on dist-test-slave-jkbq"
server_key: "fbcdbb5a4fa74e020774ff606d6ce4cc"
server_key_iv: "be100d16176d2327559988da02678b5f"
server_key_version: "encryptionkey@0"
I20260620 23:12:19.122438 21430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:12:19.139756 21430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:12:19.144217 21430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:12:19.144521 21430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:12:19.145473 21430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:12:19.146816 21430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:12:19.146950 21430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260620 23:12:19.147058 21430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:12:19.147150 21430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:19.180083 21430 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.193:33981
I20260620 23:12:19.180161 21557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.193:33981 every 8 connection(s)
I20260620 23:12:19.181355 21430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
I20260620 23:12:19.188117 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 21430
I20260620 23:12:19.188355 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance
I20260620 23:12:19.188974 11007 external_mini_cluster.cc:1428] Setting key d1e79170658d64282d5ed54a4746cee6
Jun 20 23:12:19 dist-test-slave-jkbq krb5kdc[21324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997139, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:12:19.201329 21560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:19.183935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:43737 (local address 127.10.191.254:33655)
0620 23:12:19.184401 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:19.184405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:19.185212 (+ 807us) server_negotiation.cc:407] Connection header received
0620 23:12:19.187033 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:19.187039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:19.187137 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:19.187260 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:19.188795 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:19.189577 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:19.191460 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:19.191865 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:19.192007 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:19.195425 (+ 3418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:19.195430 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:19.195434 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:19.195463 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:19.198125 (+ 2662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:19.199065 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:19.199069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:19.199072 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:19.199254 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:19.199731 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:19.199735 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:19.199737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:19.199978 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:19.200087 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:19.200917 (+ 830us) server_negotiation.cc:299] Negotiation successful
0620 23:12:19.201074 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":258,"threads_started":1}
I20260620 23:12:19.202823 21561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:19.184373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:33655 (local address 127.10.191.193:43737)
0620 23:12:19.185048 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:19.185089 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:19.186703 (+ 1614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:19.187495 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:19.187503 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:19.187969 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:19.188627 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:19.188645 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:19.189747 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:19.189757 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:19.191310 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:19.191321 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:19.192094 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:19.192103 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:19.192331 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:19.193010 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:19.193039 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:19.195220 (+ 2181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:12:19.198256 (+ 3036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:19.198264 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:19.198280 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:19.198939 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:19.199373 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:19.199380 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:19.199386 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:19.199620 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:19.200102 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:12:19.200113 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:12:19.200525 (+ 412us) client_negotiation.cc:769] Sending connection context
0620 23:12:19.200824 (+ 299us) client_negotiation.cc:240] Negotiation successful
0620 23:12:19.201372 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":263,"threads_started":1}
I20260620 23:12:19.204461 21558 heartbeater.cc:347] Connected to a master server at 127.10.191.254:33655
I20260620 23:12:19.204802 21558 heartbeater.cc:464] Registering TS with master...
I20260620 23:12:19.205528 21558 heartbeater.cc:511] Master 127.10.191.254:33655 requested a full tablet report, sending...
I20260620 23:12:19.207634 21369 ts_manager.cc:194] Registered new tserver with Master: ed18b2da9d8348008d41f632748f140c (127.10.191.193:33981)
I20260620 23:12:19.209091 21369 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.193@KRBTEST.COM'} at 127.10.191.193:43737
WARNING: no policy specified for kudu/127.10.191.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab.
Entry for principal kudu/127.10.191.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab.
I20260620 23:12:19.239441 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.194:0
--local_ip_for_outbound_sockets=127.10.191.194
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:33655
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:39441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:12:19.556324 21564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:12:19.556663 21564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:12:19.556821 21564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:12:19.568061 21564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:12:19.568187 21564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:12:19.568363 21564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.194
I20260620 23:12:19.580577 21564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:39441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:33655
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.194
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:12:19.581746 21564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:12:19.583091 21564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:12:19.595300 21570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:19.595333 21572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:19.595681 21569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:12:19.596061 21564 server_base.cc:1034] running on GCE node
I20260620 23:12:19.596623 21564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:12:19.597497 21564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:12:19.598722 21564 hybrid_clock.cc:648] HybridClock initialized: now 1781997139598681 us; error 57 us; skew 500 ppm
I20260620 23:12:19.599097 21564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:12:19 dist-test-slave-jkbq krb5kdc[21324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997139, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:12:19.605458 21564 init.cc:377] Logged in from keytab as kudu/127.10.191.194@KRBTEST.COM (short username kudu)
I20260620 23:12:19.606778 21564 webserver.cc:460] Webserver started at http://127.10.191.194:43609/ using document root <none> and password file <none>
I20260620 23:12:19.607565 21564 fs_manager.cc:359] Metadata directory not provided
I20260620 23:12:19.607721 21564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:12:19.608004 21564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:12:19.610679 21564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/data/instance:
uuid: "b260dc6aecc44033a89b01116b7f360b"
format_stamp: "Formatted at 2026-06-20 23:12:19 on dist-test-slave-jkbq"
server_key: "fae011d104c6fba66c51610d82c3b1aa"
server_key_iv: "5c80d82532ebe61fbeb3022d6700ebab"
server_key_version: "encryptionkey@0"
I20260620 23:12:19.611490 21564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance:
uuid: "b260dc6aecc44033a89b01116b7f360b"
format_stamp: "Formatted at 2026-06-20 23:12:19 on dist-test-slave-jkbq"
server_key: "fae011d104c6fba66c51610d82c3b1aa"
server_key_iv: "5c80d82532ebe61fbeb3022d6700ebab"
server_key_version: "encryptionkey@0"
I20260620 23:12:19.617666 21564 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 23:12:19.623106 21579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:19.624918 21564 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 23:12:19.625098 21564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/wal
uuid: "b260dc6aecc44033a89b01116b7f360b"
format_stamp: "Formatted at 2026-06-20 23:12:19 on dist-test-slave-jkbq"
server_key: "fae011d104c6fba66c51610d82c3b1aa"
server_key_iv: "5c80d82532ebe61fbeb3022d6700ebab"
server_key_version: "encryptionkey@0"
I20260620 23:12:19.625311 21564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:12:19.662955 21564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:12:19.667388 21564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:12:19.667708 21564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:12:19.668646 21564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:12:19.670056 21564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:12:19.670158 21564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:19.670307 21564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:12:19.670383 21564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:19.704730 21564 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.194:36179
I20260620 23:12:19.704771 21691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.194:36179 every 8 connection(s)
I20260620 23:12:19.706120 21564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
I20260620 23:12:19.710748 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 21564
I20260620 23:12:19.710958 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance
I20260620 23:12:19.711376 11007 external_mini_cluster.cc:1428] Setting key d0ca3bfb2eecd18c467b4b27a8e99b80
Jun 20 23:12:19 dist-test-slave-jkbq krb5kdc[21324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997139, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:12:19.728523 21694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:19.708805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:59691 (local address 127.10.191.254:33655)
0620 23:12:19.709203 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:19.709207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:19.710215 (+ 1008us) server_negotiation.cc:407] Connection header received
0620 23:12:19.712305 (+ 2090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:19.712311 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:19.712435 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:19.712548 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:19.714388 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:19.715164 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:19.717594 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:19.718004 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:19.718097 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:19.722427 (+ 4330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:19.722432 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:19.722435 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:19.722468 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:19.725273 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:19.726073 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:19.726077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:19.726079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:19.726206 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:19.726776 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:19.726787 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:19.726789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:19.727101 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:19.727214 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:19.728099 (+ 885us) server_negotiation.cc:299] Negotiation successful
0620 23:12:19.728263 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":200,"threads_started":1}
I20260620 23:12:19.730159 21695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:19.709402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:33655 (local address 127.10.191.194:59691)
0620 23:12:19.710051 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:19.710091 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:19.712005 (+ 1914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:19.712765 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:19.712773 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:19.713424 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:19.714196 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:19.714240 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:19.715321 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:19.715330 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:19.717441 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:19.717459 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:19.718237 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:19.718246 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:19.718562 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:19.719315 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:19.719349 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:19.722185 (+ 2836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:12:19.725419 (+ 3234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:19.725429 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:19.725446 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:19.725941 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:19.726350 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:19.726358 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:19.726364 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:19.726632 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:19.727229 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:12:19.727240 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:12:19.727711 (+ 471us) client_negotiation.cc:769] Sending connection context
0620 23:12:19.728028 (+ 317us) client_negotiation.cc:240] Negotiation successful
0620 23:12:19.728633 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":258,"threads_started":1}
I20260620 23:12:19.731822 21692 heartbeater.cc:347] Connected to a master server at 127.10.191.254:33655
I20260620 23:12:19.732177 21692 heartbeater.cc:464] Registering TS with master...
I20260620 23:12:19.732925 21692 heartbeater.cc:511] Master 127.10.191.254:33655 requested a full tablet report, sending...
I20260620 23:12:19.734391 21369 ts_manager.cc:194] Registered new tserver with Master: b260dc6aecc44033a89b01116b7f360b (127.10.191.194:36179)
I20260620 23:12:19.735273 21369 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.194@KRBTEST.COM'} at 127.10.191.194:59691
WARNING: no policy specified for kudu/127.10.191.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab.
Entry for principal kudu/127.10.191.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab.
I20260620 23:12:19.764343 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.195:0
--local_ip_for_outbound_sockets=127.10.191.195
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:33655
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:39441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:12:20.084424 21698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:12:20.084681 21698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:12:20.084859 21698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:12:20.091758 21698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:12:20.091857 21698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:12:20.092074 21698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.195
I20260620 23:12:20.101477 21698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:39441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:33655
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.195
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:12:20.102735 21698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:12:20.104012 21698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:12:20.116786 21703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:20.116815 21704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:12:20.116950 21706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:12:20.117537 21698 server_base.cc:1034] running on GCE node
I20260620 23:12:20.118165 21698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:12:20.119104 21698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:12:20.120294 21698 hybrid_clock.cc:648] HybridClock initialized: now 1781997140120257 us; error 27 us; skew 500 ppm
I20260620 23:12:20.120592 21698 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:12:20 dist-test-slave-jkbq krb5kdc[21324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997140, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:12:20.123700 21698 init.cc:377] Logged in from keytab as kudu/127.10.191.195@KRBTEST.COM (short username kudu)
I20260620 23:12:20.125030 21698 webserver.cc:460] Webserver started at http://127.10.191.195:36247/ using document root <none> and password file <none>
I20260620 23:12:20.125675 21698 fs_manager.cc:359] Metadata directory not provided
I20260620 23:12:20.125811 21698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:12:20.126133 21698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:12:20.128959 21698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/data/instance:
uuid: "a2ea8025a3f241ca80f437b8d2cc79cb"
format_stamp: "Formatted at 2026-06-20 23:12:20 on dist-test-slave-jkbq"
server_key: "ef88c1ff2c4c04b89dd52335091fd26f"
server_key_iv: "5446a1b5514cac796c0783a4d717533a"
server_key_version: "encryptionkey@0"
I20260620 23:12:20.129719 21698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance:
uuid: "a2ea8025a3f241ca80f437b8d2cc79cb"
format_stamp: "Formatted at 2026-06-20 23:12:20 on dist-test-slave-jkbq"
server_key: "ef88c1ff2c4c04b89dd52335091fd26f"
server_key_iv: "5446a1b5514cac796c0783a4d717533a"
server_key_version: "encryptionkey@0"
I20260620 23:12:20.134728 21698 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 23:12:20.137926 21713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:20.139258 21698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:12:20.139418 21698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/wal
uuid: "a2ea8025a3f241ca80f437b8d2cc79cb"
format_stamp: "Formatted at 2026-06-20 23:12:20 on dist-test-slave-jkbq"
server_key: "ef88c1ff2c4c04b89dd52335091fd26f"
server_key_iv: "5446a1b5514cac796c0783a4d717533a"
server_key_version: "encryptionkey@0"
I20260620 23:12:20.139596 21698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:12:20.160557 21698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:12:20.164525 21698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:12:20.164824 21698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:12:20.165793 21698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:12:20.167079 21698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:12:20.167150 21698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:20.167295 21698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:12:20.167346 21698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:12:20.200047 21698 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.195:41449
I20260620 23:12:20.200093 21825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.195:41449 every 8 connection(s)
I20260620 23:12:20.201292 21698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
I20260620 23:12:20.204880 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 21698
I20260620 23:12:20.205075 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance
I20260620 23:12:20.205511 11007 external_mini_cluster.cc:1428] Setting key c5a2ebd506662e92b7ff091f2335f845
I20260620 23:12:20.212008 21558 heartbeater.cc:503] Master 127.10.191.254:33655 was elected leader, sending a full tablet report...
Jun 20 23:12:20 dist-test-slave-jkbq krb5kdc[21324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997140, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:12:20.221299 21694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:20.203786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:37855 (local address 127.10.191.254:33655)
0620 23:12:20.203958 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:20.203963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:20.205017 (+ 1054us) server_negotiation.cc:407] Connection header received
0620 23:12:20.207176 (+ 2159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:20.207182 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:20.207283 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:20.207442 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:20.209194 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:20.209932 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:20.211445 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:20.211826 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:20.211919 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:20.215527 (+ 3608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:20.215531 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:20.215535 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:20.215565 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:20.218132 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:20.218812 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:20.218816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:20.218818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:20.218990 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:20.219387 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:20.219394 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:20.219397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:20.219769 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:20.219952 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:20.220821 (+ 869us) server_negotiation.cc:299] Negotiation successful
0620 23:12:20.221062 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 23:12:20.222378 21828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:20.204172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:33655 (local address 127.10.191.195:37855)
0620 23:12:20.204856 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:20.204893 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:20.206911 (+ 2018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:20.207716 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:20.207726 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:20.208453 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:12:20.209045 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:20.209066 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:20.210049 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:20.210052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:20.211329 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:20.211339 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:20.212032 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:20.212042 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:20.212323 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:20.212991 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:12:20.213020 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:12:20.215349 (+ 2329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:12:20.218274 (+ 2925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:20.218280 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:20.218293 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:20.218688 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:20.219095 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:12:20.219100 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:12:20.219103 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:12:20.219293 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:12:20.219909 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:12:20.219917 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:12:20.220376 (+ 459us) client_negotiation.cc:769] Sending connection context
0620 23:12:20.220671 (+ 295us) client_negotiation.cc:240] Negotiation successful
0620 23:12:20.221184 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":278,"threads_started":1}
I20260620 23:12:20.223702 21826 heartbeater.cc:347] Connected to a master server at 127.10.191.254:33655
I20260620 23:12:20.224014 21826 heartbeater.cc:464] Registering TS with master...
I20260620 23:12:20.224735 21826 heartbeater.cc:511] Master 127.10.191.254:33655 requested a full tablet report, sending...
I20260620 23:12:20.226171 21369 ts_manager.cc:194] Registered new tserver with Master: a2ea8025a3f241ca80f437b8d2cc79cb (127.10.191.195:41449)
I20260620 23:12:20.227028 21369 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.195@KRBTEST.COM'} at 127.10.191.195:37855
I20260620 23:12:20.232996 11007 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:12:20.248932 21694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:20.237294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58558 (local address 127.10.191.254:33655)
0620 23:12:20.237452 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:20.237456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:20.238014 (+ 558us) server_negotiation.cc:407] Connection header received
0620 23:12:20.238148 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:20.238152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:20.238226 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:20.238303 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:20.239714 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:20.240530 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:20.241677 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:20.242046 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:20.242133 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:20.244488 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:20.244493 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:20.244496 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:20.244524 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:20.246760 (+ 2236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:20.247382 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:20.247386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:20.247388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:20.247505 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:20.247826 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:20.247829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:20.247831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:20.248105 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:20.248289 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:20.248559 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 23:12:20.248704 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 23:12:20.252887 21369 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:12:20.259666 21369 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:12:20.298343 21841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:20.272239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39710 (local address 127.10.191.195:41449)
0620 23:12:20.289386 (+ 17147us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:20.289398 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:20.289440 (+ 42us) server_negotiation.cc:407] Connection header received
0620 23:12:20.289561 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:20.289572 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:20.289860 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:20.290095 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:12:20.291155 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:20.292295 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:20.295643 (+ 3348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:20.296982 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:20.297162 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:20.297310 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:20.297851 (+ 541us) server_negotiation.cc:299] Negotiation successful
0620 23:12:20.298014 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16949,"thread_start_us":16784,"threads_started":1}
I20260620 23:12:20.298393 21842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:20.271636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.195:41449 (local address 127.0.0.1:39710)
0620 23:12:20.288482 (+ 16846us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:20.288502 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:20.288650 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:20.290405 (+ 1755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:20.290410 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:20.290438 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:12:20.290938 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:20.290950 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:20.292489 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:20.292495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:20.295486 (+ 2991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:20.295501 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:20.297450 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:20.297455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:20.297672 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:20.297697 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 23:12:20.297789 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 23:12:20.297942 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16700,"mutex_wait_us":87,"thread_start_us":1332,"threads_started":1}
I20260620 23:12:20.298686 21839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:20.271590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48594 (local address 127.10.191.194:36179)
0620 23:12:20.289137 (+ 17547us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:20.289147 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:20.289180 (+ 33us) server_negotiation.cc:407] Connection header received
0620 23:12:20.289297 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:20.289306 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:20.289639 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:20.290439 (+ 800us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:12:20.290796 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:20.292313 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:20.295423 (+ 3110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:20.296942 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:20.297129 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:20.297310 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:20.298052 (+ 742us) server_negotiation.cc:299] Negotiation successful
0620 23:12:20.298219 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17360,"thread_start_us":18147,"threads_started":1}
I20260620 23:12:20.298983 21840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:20.271368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.194:36179 (local address 127.0.0.1:48594)
0620 23:12:20.273346 (+ 1978us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:20.273388 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:20.273564 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:20.289919 (+ 16355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:20.289929 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:20.289969 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:12:20.290565 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:20.290580 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:20.292448 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:20.292457 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:20.295276 (+ 2819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:20.295287 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:20.297164 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:20.297170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:20.297363 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:20.297427 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 23:12:20.297557 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 23:12:20.297750 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1739,"mutex_wait_us":518,"thread_start_us":18760,"threads_started":1}
I20260620 23:12:20.299448 21843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:20.274184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55042 (local address 127.10.191.193:33981)
0620 23:12:20.282998 (+ 8814us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:20.283013 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:20.290591 (+ 7578us) server_negotiation.cc:407] Connection header received
0620 23:12:20.290926 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:20.290936 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:20.291171 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:20.291328 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:12:20.292169 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:20.293807 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:20.296315 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:20.297389 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:20.297617 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:20.297770 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:20.298386 (+ 616us) server_negotiation.cc:299] Negotiation successful
0620 23:12:20.298909 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7994,"thread_start_us":7360,"threads_started":1}
I20260620 23:12:20.299860 21838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:20.273766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:33981 (local address 127.0.0.1:55042)
0620 23:12:20.290455 (+ 16689us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:20.290479 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:20.290701 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:20.291616 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:20.291621 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:20.291644 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:12:20.291972 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:20.291982 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:20.293977 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:20.293983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:20.296175 (+ 2192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:20.296186 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:20.297922 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:20.297930 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:20.298146 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:20.298208 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 23:12:20.299274 (+ 1066us) client_negotiation.cc:240] Negotiation successful
0620 23:12:20.299407 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16542,"mutex_wait_us":22,"thread_start_us":14409,"threads_started":1}
I20260620 23:12:20.302350 21761 tablet_service.cc:1467] Processing CreateTablet for tablet 10adf11deaa641af965b16c2389ffdcc (DEFAULT_TABLE table=test-table [id=f33d62894d754b50b064303336dba0c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:12:20.302352 21627 tablet_service.cc:1467] Processing CreateTablet for tablet 10adf11deaa641af965b16c2389ffdcc (DEFAULT_TABLE table=test-table [id=f33d62894d754b50b064303336dba0c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:12:20.303601 21761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10adf11deaa641af965b16c2389ffdcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:12:20.303614 21627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10adf11deaa641af965b16c2389ffdcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:12:20.309979 21493 tablet_service.cc:1467] Processing CreateTablet for tablet 10adf11deaa641af965b16c2389ffdcc (DEFAULT_TABLE table=test-table [id=f33d62894d754b50b064303336dba0c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:12:20.312247 21493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10adf11deaa641af965b16c2389ffdcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:12:20.315595 21844 tablet_bootstrap.cc:492] T 10adf11deaa641af965b16c2389ffdcc P b260dc6aecc44033a89b01116b7f360b: Bootstrap starting.
I20260620 23:12:20.318616 21844 tablet_bootstrap.cc:654] T 10adf11deaa641af965b16c2389ffdcc P b260dc6aecc44033a89b01116b7f360b: Neither blocks nor log segments found. Creating new log.
I20260620 23:12:20.319502 21845 tablet_bootstrap.cc:492] T 10adf11deaa641af965b16c2389ffdcc P a2ea8025a3f241ca80f437b8d2cc79cb: Bootstrap starting.
I20260620 23:12:20.320006 21844 log.cc:824] T 10adf11deaa641af965b16c2389ffdcc P b260dc6aecc44033a89b01116b7f360b: Log is configured to *not* fsync() on all Append() calls
I20260620 23:12:20.324113 21845 tablet_bootstrap.cc:654] T 10adf11deaa641af965b16c2389ffdcc P a2ea8025a3f241ca80f437b8d2cc79cb: Neither blocks nor log segments found. Creating new log.
I20260620 23:12:20.325376 21845 log.cc:824] T 10adf11deaa641af965b16c2389ffdcc P a2ea8025a3f241ca80f437b8d2cc79cb: Log is configured to *not* fsync() on all Append() calls
I20260620 23:12:20.327095 21844 tablet_bootstrap.cc:492] T 10adf11deaa641af965b16c2389ffdcc P b260dc6aecc44033a89b01116b7f360b: No bootstrap required, opened a new log
I20260620 23:12:20.327380 21844 ts_tablet_manager.cc:1397] T 10adf11deaa641af965b16c2389ffdcc P b260dc6aecc44033a89b01116b7f360b: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.005s
I20260620 23:12:20.327636 21848 tablet_bootstrap.cc:492] T 10adf11deaa641af965b16c2389ffdcc P ed18b2da9d8348008d41f632748f140c: Bootstrap starting.
I20260620 23:12:20.328501 21845 tablet_bootstrap.cc:492] T 10adf11deaa641af965b16c2389ffdcc P a2ea8025a3f241ca80f437b8d2cc79cb: No bootstrap required, opened a new log
I20260620 23:12:20.328768 21845 ts_tablet_manager.cc:1397] T 10adf11deaa641af965b16c2389ffdcc P a2ea8025a3f241ca80f437b8d2cc79cb: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260620 23:12:20.330436 21848 tablet_bootstrap.cc:654] T 10adf11deaa641af965b16c2389ffdcc P ed18b2da9d8348008d41f632748f140c: Neither blocks nor log segments found. Creating new log.
I20260620 23:12:20.331591 21848 log.cc:824] T 10adf11deaa641af965b16c2389ffdcc P ed18b2da9d8348008d41f632748f140c: Log is configured to *not* fsync() on all Append() calls
I20260620 23:12:20.334263 21848 tablet_bootstrap.cc:492] T 10adf11deaa641af965b16c2389ffdcc P ed18b2da9d8348008d41f632748f140c: No bootstrap required, opened a new log
I20260620 23:12:20.334515 21848 ts_tablet_manager.cc:1397] T 10adf11deaa641af965b16c2389ffdcc P ed18b2da9d8348008d41f632748f140c: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 23:12:20.338352 21844 raft_consensus.cc:348] T 10adf11deaa641af965b16c2389ffdcc P b260dc6aecc44033a89b01116b7f360b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b260dc6aecc44033a89b01116b7f360b" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 36179 } } peers { permanent_uuid: "a2ea8025a3f241ca80f437b8d2cc79cb" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 41449 } } peers { permanent_uuid: "ed18b2da9d8348008d41f632748f140c" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 33981 } }
I20260620 23:12:20.338810 21844 raft_consensus.cc:374] T 10adf11deaa641af965b16c2389ffdcc P b260dc6aecc44033a89b01116b7f360b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:12:20.338940 21844 raft_consensus.cc:729] T 10adf11deaa641af965b16c2389ffdcc P b260dc6aecc44033a89b01116b7f360b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b260dc6aecc44033a89b01116b7f360b, State: Initialized, Role: FOLLOWER
I20260620 23:12:20.339315 21845 raft_consensus.cc:348] T 10adf11deaa641af965b16c2389ffdcc P a2ea8025a3f241ca80f437b8d2cc79cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b260dc6aecc44033a89b01116b7f360b" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 36179 } } peers { permanent_uuid: "a2ea8025a3f241ca80f437b8d2cc79cb" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 41449 } } peers { permanent_uuid: "ed18b2da9d8348008d41f632748f140c" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 33981 } }
I20260620 23:12:20.339759 21845 raft_consensus.cc:374] T 10adf11deaa641af965b16c2389ffdcc P a2ea8025a3f241ca80f437b8d2cc79cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:12:20.339632 21844 consensus_queue.cc:260] T 10adf11deaa641af965b16c2389ffdcc P b260dc6aecc44033a89b01116b7f360b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b260dc6aecc44033a89b01116b7f360b" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 36179 } } peers { permanent_uuid: "a2ea8025a3f241ca80f437b8d2cc79cb" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 41449 } } peers { permanent_uuid: "ed18b2da9d8348008d41f632748f140c" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 33981 } }
I20260620 23:12:20.339855 21845 raft_consensus.cc:729] T 10adf11deaa641af965b16c2389ffdcc P a2ea8025a3f241ca80f437b8d2cc79cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2ea8025a3f241ca80f437b8d2cc79cb, State: Initialized, Role: FOLLOWER
I20260620 23:12:20.340531 21845 consensus_queue.cc:260] T 10adf11deaa641af965b16c2389ffdcc P a2ea8025a3f241ca80f437b8d2cc79cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b260dc6aecc44033a89b01116b7f360b" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 36179 } } peers { permanent_uuid: "a2ea8025a3f241ca80f437b8d2cc79cb" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 41449 } } peers { permanent_uuid: "ed18b2da9d8348008d41f632748f140c" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 33981 } }
I20260620 23:12:20.341807 21826 heartbeater.cc:503] Master 127.10.191.254:33655 was elected leader, sending a full tablet report...
I20260620 23:12:20.342304 21848 raft_consensus.cc:348] T 10adf11deaa641af965b16c2389ffdcc P ed18b2da9d8348008d41f632748f140c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b260dc6aecc44033a89b01116b7f360b" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 36179 } } peers { permanent_uuid: "a2ea8025a3f241ca80f437b8d2cc79cb" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 41449 } } peers { permanent_uuid: "ed18b2da9d8348008d41f632748f140c" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 33981 } }
I20260620 23:12:20.342537 21845 ts_tablet_manager.cc:1428] T 10adf11deaa641af965b16c2389ffdcc P a2ea8025a3f241ca80f437b8d2cc79cb: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260620 23:12:20.342664 21848 raft_consensus.cc:374] T 10adf11deaa641af965b16c2389ffdcc P ed18b2da9d8348008d41f632748f140c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:12:20.342783 21848 raft_consensus.cc:729] T 10adf11deaa641af965b16c2389ffdcc P ed18b2da9d8348008d41f632748f140c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed18b2da9d8348008d41f632748f140c, State: Initialized, Role: FOLLOWER
I20260620 23:12:20.343458 21848 consensus_queue.cc:260] T 10adf11deaa641af965b16c2389ffdcc P ed18b2da9d8348008d41f632748f140c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b260dc6aecc44033a89b01116b7f360b" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 36179 } } peers { permanent_uuid: "a2ea8025a3f241ca80f437b8d2cc79cb" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 41449 } } peers { permanent_uuid: "ed18b2da9d8348008d41f632748f140c" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 33981 } }
I20260620 23:12:20.344425 21692 heartbeater.cc:503] Master 127.10.191.254:33655 was elected leader, sending a full tablet report...
I20260620 23:12:20.345084 21844 ts_tablet_manager.cc:1428] T 10adf11deaa641af965b16c2389ffdcc P b260dc6aecc44033a89b01116b7f360b: Time spent starting tablet: real 0.018s user 0.013s sys 0.003s
I20260620 23:12:20.346758 21848 ts_tablet_manager.cc:1428] T 10adf11deaa641af965b16c2389ffdcc P ed18b2da9d8348008d41f632748f140c: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260620 23:12:20.369987 21851 raft_consensus.cc:482] T 10adf11deaa641af965b16c2389ffdcc P a2ea8025a3f241ca80f437b8d2cc79cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:12:20.370314 21851 raft_consensus.cc:504] T 10adf11deaa641af965b16c2389ffdcc P a2ea8025a3f241ca80f437b8d2cc79cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b260dc6aecc44033a89b01116b7f360b" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 36179 } } peers { permanent_uuid: "a2ea8025a3f241ca80f437b8d2cc79cb" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 41449 } } peers { permanent_uuid: "ed18b2da9d8348008d41f632748f140c" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 33981 } }
I20260620 23:12:20.372112 21851 leader_election.cc:288] T 10adf11deaa641af965b16c2389ffdcc P a2ea8025a3f241ca80f437b8d2cc79cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b260dc6aecc44033a89b01116b7f360b (127.10.191.194:36179), ed18b2da9d8348008d41f632748f140c (127.10.191.193:33981)
I20260620 23:12:20.378284 21843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:20.372591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:60179 (local address 127.10.191.193:33981)
0620 23:12:20.372806 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:20.372812 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:20.372833 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:12:20.373023 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:20.373031 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:20.373121 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:20.373233 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:12:20.373674 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:20.374661 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:20.376467 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:20.377515 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:20.377629 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:20.377677 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:20.378034 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 23:12:20.378115 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 23:12:20.378396 21828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:20.372508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:33981 (local address 127.10.191.195:60179)
0620 23:12:20.372681 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:20.372704 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:20.372842 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:20.373275 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:20.373283 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:20.373323 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:12:20.373500 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:20.373509 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:20.374791 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:20.374797 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:20.376312 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:20.376324 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:20.377662 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:20.377669 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:20.377866 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:20.377902 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 23:12:20.378001 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 23:12:20.378133 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 23:12:20.379343 21839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:20.372664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:53023 (local address 127.10.191.194:36179)
0620 23:12:20.372841 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:20.372846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:20.373584 (+ 738us) server_negotiation.cc:407] Connection header received
0620 23:12:20.373713 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:20.373717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:20.373771 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:20.373837 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:12:20.374382 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:20.375334 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:20.377230 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:20.378283 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:20.378426 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:20.378496 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:20.378977 (+ 481us) server_negotiation.cc:299] Negotiation successful
0620 23:12:20.379098 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 23:12:20.379451 21853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:20.372519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.194:36179 (local address 127.10.191.195:53023)
0620 23:12:20.373466 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0620 23:12:20.373480 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:12:20.373581 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:12:20.373873 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:12:20.373878 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:12:20.373901 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:12:20.374238 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:20.374249 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:20.375467 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:20.375471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:20.377069 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:12:20.377083 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:12:20.378655 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:12:20.378661 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:12:20.378810 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:20.378832 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 23:12:20.378952 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 23:12:20.379073 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":830,"mutex_wait_us":197,"thread_start_us":384,"threads_started":1}
I20260620 23:12:20.379376 21513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10adf11deaa641af965b16c2389ffdcc" candidate_uuid: "a2ea8025a3f241ca80f437b8d2cc79cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed18b2da9d8348008d41f632748f140c" is_pre_election: true
I20260620 23:12:20.380012 21513 raft_consensus.cc:2454] T 10adf11deaa641af965b16c2389ffdcc P ed18b2da9d8348008d41f632748f140c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2ea8025a3f241ca80f437b8d2cc79cb in term 0.
I20260620 23:12:20.380338 21647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10adf11deaa641af965b16c2389ffdcc" candidate_uuid: "a2ea8025a3f241ca80f437b8d2cc79cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b260dc6aecc44033a89b01116b7f360b" is_pre_election: true
I20260620 23:12:20.380683 21714 leader_election.cc:302] T 10adf11deaa641af965b16c2389ffdcc P a2ea8025a3f241ca80f437b8d2cc79cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2ea8025a3f241ca80f437b8d2cc79cb, ed18b2da9d8348008d41f632748f140c; no voters:
I20260620 23:12:20.380769 21647 raft_consensus.cc:2454] T 10adf11deaa641af965b16c2389ffdcc P b260dc6aecc44033a89b01116b7f360b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2ea8025a3f241ca80f437b8d2cc79cb in term 0.
I20260620 23:12:20.381060 21851 raft_consensus.cc:2781] T 10adf11deaa641af965b16c2389ffdcc P a2ea8025a3f241ca80f437b8d2cc79cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:12:20.381215 21851 raft_consensus.cc:482] T 10adf11deaa641af965b16c2389ffdcc P a2ea8025a3f241ca80f437b8d2cc79cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:12:20.381353 21851 raft_consensus.cc:3037] T 10adf11deaa641af965b16c2389ffdcc P a2ea8025a3f241ca80f437b8d2cc79cb [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:12:20.383653 21851 raft_consensus.cc:504] T 10adf11deaa641af965b16c2389ffdcc P a2ea8025a3f241ca80f437b8d2cc79cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b260dc6aecc44033a89b01116b7f360b" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 36179 } } peers { permanent_uuid: "a2ea8025a3f241ca80f437b8d2cc79cb" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 41449 } } peers { permanent_uuid: "ed18b2da9d8348008d41f632748f140c" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 33981 } }
I20260620 23:12:20.384276 21851 leader_election.cc:288] T 10adf11deaa641af965b16c2389ffdcc P a2ea8025a3f241ca80f437b8d2cc79cb [CANDIDATE]: Term 1 election: Requested vote from peers b260dc6aecc44033a89b01116b7f360b (127.10.191.194:36179), ed18b2da9d8348008d41f632748f140c (127.10.191.193:33981)
I20260620 23:12:20.384699 21647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10adf11deaa641af965b16c2389ffdcc" candidate_uuid: "a2ea8025a3f241ca80f437b8d2cc79cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b260dc6aecc44033a89b01116b7f360b"
I20260620 23:12:20.384918 21647 raft_consensus.cc:3037] T 10adf11deaa641af965b16c2389ffdcc P b260dc6aecc44033a89b01116b7f360b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:12:20.384883 21513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10adf11deaa641af965b16c2389ffdcc" candidate_uuid: "a2ea8025a3f241ca80f437b8d2cc79cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed18b2da9d8348008d41f632748f140c"
I20260620 23:12:20.385118 21513 raft_consensus.cc:3037] T 10adf11deaa641af965b16c2389ffdcc P ed18b2da9d8348008d41f632748f140c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:12:20.387184 21647 raft_consensus.cc:2454] T 10adf11deaa641af965b16c2389ffdcc P b260dc6aecc44033a89b01116b7f360b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2ea8025a3f241ca80f437b8d2cc79cb in term 1.
I20260620 23:12:20.387297 21513 raft_consensus.cc:2454] T 10adf11deaa641af965b16c2389ffdcc P ed18b2da9d8348008d41f632748f140c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2ea8025a3f241ca80f437b8d2cc79cb in term 1.
I20260620 23:12:20.387629 21716 leader_election.cc:302] T 10adf11deaa641af965b16c2389ffdcc P a2ea8025a3f241ca80f437b8d2cc79cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2ea8025a3f241ca80f437b8d2cc79cb, b260dc6aecc44033a89b01116b7f360b; no voters:
I20260620 23:12:20.387916 21851 raft_consensus.cc:2781] T 10adf11deaa641af965b16c2389ffdcc P a2ea8025a3f241ca80f437b8d2cc79cb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:12:20.388327 21851 raft_consensus.cc:686] T 10adf11deaa641af965b16c2389ffdcc P a2ea8025a3f241ca80f437b8d2cc79cb [term 1 LEADER]: Becoming Leader. State: Replica: a2ea8025a3f241ca80f437b8d2cc79cb, State: Running, Role: LEADER
I20260620 23:12:20.388903 21851 consensus_queue.cc:237] T 10adf11deaa641af965b16c2389ffdcc P a2ea8025a3f241ca80f437b8d2cc79cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b260dc6aecc44033a89b01116b7f360b" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 36179 } } peers { permanent_uuid: "a2ea8025a3f241ca80f437b8d2cc79cb" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 41449 } } peers { permanent_uuid: "ed18b2da9d8348008d41f632748f140c" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 33981 } }
I20260620 23:12:20.393814 21367 catalog_manager.cc:5495] T 10adf11deaa641af965b16c2389ffdcc P a2ea8025a3f241ca80f437b8d2cc79cb reported cstate change: term changed from 0 to 1, leader changed from <none> to a2ea8025a3f241ca80f437b8d2cc79cb (127.10.191.195). New cstate: current_term: 1 leader_uuid: "a2ea8025a3f241ca80f437b8d2cc79cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b260dc6aecc44033a89b01116b7f360b" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 36179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2ea8025a3f241ca80f437b8d2cc79cb" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 41449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed18b2da9d8348008d41f632748f140c" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 33981 } health_report { overall_health: UNKNOWN } } }
W20260620 23:12:20.433673 21559 tablet.cc:2369] T 10adf11deaa641af965b16c2389ffdcc P ed18b2da9d8348008d41f632748f140c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:12:20.436264 21841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:20.431147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39726 (local address 127.10.191.195:41449)
0620 23:12:20.431396 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:20.431401 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:20.431419 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:12:20.431553 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:20.431557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:20.431639 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:20.431764 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:12:20.432081 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:20.432784 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:20.434128 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:20.434660 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:20.434836 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:20.435283 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:12:20.435791 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:12:20.435939 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:20.436010 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 23:12:20.436074 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 23:12:20.444216 21513 raft_consensus.cc:1261] T 10adf11deaa641af965b16c2389ffdcc P ed18b2da9d8348008d41f632748f140c [term 1 FOLLOWER]: Refusing update from remote peer a2ea8025a3f241ca80f437b8d2cc79cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:12:20.444242 21647 raft_consensus.cc:1261] T 10adf11deaa641af965b16c2389ffdcc P b260dc6aecc44033a89b01116b7f360b [term 1 FOLLOWER]: Refusing update from remote peer a2ea8025a3f241ca80f437b8d2cc79cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:12:20.445190 21851 consensus_queue.cc:1035] T 10adf11deaa641af965b16c2389ffdcc P a2ea8025a3f241ca80f437b8d2cc79cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed18b2da9d8348008d41f632748f140c" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 33981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:12:20.445468 21854 consensus_queue.cc:1035] T 10adf11deaa641af965b16c2389ffdcc P a2ea8025a3f241ca80f437b8d2cc79cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b260dc6aecc44033a89b01116b7f360b" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 36179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 23:12:20.452893 21827 tablet.cc:2369] T 10adf11deaa641af965b16c2389ffdcc P a2ea8025a3f241ca80f437b8d2cc79cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:12:20.472167 21367 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:12:20.472427 21367 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:12:20.475975 21367 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 94a04d2d407a4a8aa0df53e94e3a4d3a: Sending DeleteTablet for 3 replicas of tablet 10adf11deaa641af965b16c2389ffdcc
I20260620 23:12:20.477155 21627 tablet_service.cc:1514] Processing DeleteTablet for tablet 10adf11deaa641af965b16c2389ffdcc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:12:20 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:48594
I20260620 23:12:20.477169 21761 tablet_service.cc:1514] Processing DeleteTablet for tablet 10adf11deaa641af965b16c2389ffdcc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:12:20 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:39710
I20260620 23:12:20.477387 21493 tablet_service.cc:1514] Processing DeleteTablet for tablet 10adf11deaa641af965b16c2389ffdcc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:12:20 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:55042
I20260620 23:12:20.478161 21867 tablet_replica.cc:330] T 10adf11deaa641af965b16c2389ffdcc P a2ea8025a3f241ca80f437b8d2cc79cb: stopping tablet replica
I20260620 23:12:20.478720 21867 raft_consensus.cc:2229] T 10adf11deaa641af965b16c2389ffdcc P a2ea8025a3f241ca80f437b8d2cc79cb [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:12:20.479357 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 21430
I20260620 23:12:20.479637 21867 raft_consensus.cc:2258] T 10adf11deaa641af965b16c2389ffdcc P a2ea8025a3f241ca80f437b8d2cc79cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:12:20.480731 21868 tablet_replica.cc:330] T 10adf11deaa641af965b16c2389ffdcc P b260dc6aecc44033a89b01116b7f360b: stopping tablet replica
I20260620 23:12:20.481320 21868 raft_consensus.cc:2229] T 10adf11deaa641af965b16c2389ffdcc P b260dc6aecc44033a89b01116b7f360b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:12:20.481948 21868 raft_consensus.cc:2258] T 10adf11deaa641af965b16c2389ffdcc P b260dc6aecc44033a89b01116b7f360b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:12:20.484004 21867 ts_tablet_manager.cc:1905] T 10adf11deaa641af965b16c2389ffdcc P a2ea8025a3f241ca80f437b8d2cc79cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:12:20.484673 21868 ts_tablet_manager.cc:1905] T 10adf11deaa641af965b16c2389ffdcc P b260dc6aecc44033a89b01116b7f360b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:12:20.492879 21868 ts_tablet_manager.cc:1918] T 10adf11deaa641af965b16c2389ffdcc P b260dc6aecc44033a89b01116b7f360b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:12:20.493135 21868 log.cc:1196] T 10adf11deaa641af965b16c2389ffdcc P b260dc6aecc44033a89b01116b7f360b: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-1/wal/wals/10adf11deaa641af965b16c2389ffdcc
I20260620 23:12:20.493705 21868 ts_tablet_manager.cc:1939] T 10adf11deaa641af965b16c2389ffdcc P b260dc6aecc44033a89b01116b7f360b: Deleting consensus metadata
I20260620 23:12:20.495728 21355 catalog_manager.cc:4841] TS b260dc6aecc44033a89b01116b7f360b (127.10.191.194:36179): tablet 10adf11deaa641af965b16c2389ffdcc (table test-table [id=f33d62894d754b50b064303336dba0c3]) successfully deleted
I20260620 23:12:20.496533 21869 tablet_replica.cc:330] T 10adf11deaa641af965b16c2389ffdcc P ed18b2da9d8348008d41f632748f140c: stopping tablet replica
I20260620 23:12:20.496765 21867 ts_tablet_manager.cc:1918] T 10adf11deaa641af965b16c2389ffdcc P a2ea8025a3f241ca80f437b8d2cc79cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:12:20.496958 21867 log.cc:1196] T 10adf11deaa641af965b16c2389ffdcc P a2ea8025a3f241ca80f437b8d2cc79cb: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-2/wal/wals/10adf11deaa641af965b16c2389ffdcc
I20260620 23:12:20.496977 21869 raft_consensus.cc:2229] T 10adf11deaa641af965b16c2389ffdcc P ed18b2da9d8348008d41f632748f140c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:12:20.497465 21869 raft_consensus.cc:2258] T 10adf11deaa641af965b16c2389ffdcc P ed18b2da9d8348008d41f632748f140c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:12:20.497547 21867 ts_tablet_manager.cc:1939] T 10adf11deaa641af965b16c2389ffdcc P a2ea8025a3f241ca80f437b8d2cc79cb: Deleting consensus metadata
I20260620 23:12:20.499276 21353 catalog_manager.cc:4841] TS a2ea8025a3f241ca80f437b8d2cc79cb (127.10.191.195:41449): tablet 10adf11deaa641af965b16c2389ffdcc (table test-table [id=f33d62894d754b50b064303336dba0c3]) successfully deleted
Jun 20 23:12:20 dist-test-slave-jkbq krb5kdc[21324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.193@KRBTEST.COM
I20260620 23:12:20.502898 21869 ts_tablet_manager.cc:1905] T 10adf11deaa641af965b16c2389ffdcc P ed18b2da9d8348008d41f632748f140c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:12:20.508730 21869 ts_tablet_manager.cc:1918] T 10adf11deaa641af965b16c2389ffdcc P ed18b2da9d8348008d41f632748f140c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:12:20.508899 21869 log.cc:1196] T 10adf11deaa641af965b16c2389ffdcc P ed18b2da9d8348008d41f632748f140c: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781997042893763-11007-0/minicluster-data/ts-0/wal/wals/10adf11deaa641af965b16c2389ffdcc
I20260620 23:12:20.509471 21869 ts_tablet_manager.cc:1939] T 10adf11deaa641af965b16c2389ffdcc P ed18b2da9d8348008d41f632748f140c: Deleting consensus metadata
I20260620 23:12:20.511112 21843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:20.481567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55046 (local address 127.10.191.193:33981)
0620 23:12:20.481759 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:20.481766 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:20.490076 (+ 8310us) server_negotiation.cc:407] Connection header received
0620 23:12:20.490538 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:20.490545 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:20.490638 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:20.490709 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:20.492506 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:20.493777 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:20.497878 (+ 4101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:20.498395 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:20.498472 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:20.503281 (+ 4809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:20.503297 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:20.503316 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:20.503367 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:20.507485 (+ 4118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:20.508337 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:20.508348 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:20.508354 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:20.508527 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:20.509038 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:20.509046 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:20.509053 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:20.509539 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:20.509866 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:20.510437 (+ 571us) server_negotiation.cc:299] Negotiation successful
0620 23:12:20.510690 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 23:12:20.511600 21353 catalog_manager.cc:4841] TS ed18b2da9d8348008d41f632748f140c (127.10.191.193:33981): tablet 10adf11deaa641af965b16c2389ffdcc (table test-table [id=f33d62894d754b50b064303336dba0c3]) successfully deleted
I20260620 23:12:20.512084 21553 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:12:20.603521 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 21430
I20260620 23:12:20.621452 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 21564
Jun 20 23:12:20 dist-test-slave-jkbq krb5kdc[21324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.194@KRBTEST.COM
I20260620 23:12:20.639421 21839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:20.622342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48598 (local address 127.10.191.194:36179)
0620 23:12:20.622540 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:20.622546 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:20.622566 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:12:20.622752 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:20.622758 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:20.622833 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:20.622985 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:20.624509 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:20.625270 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:20.627131 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:20.627576 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:20.627674 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:20.632543 (+ 4869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:20.632559 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:20.632574 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:20.632610 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:20.636525 (+ 3915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:20.637275 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:20.637284 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:20.637288 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:20.637423 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:20.637798 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:20.637805 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:20.637808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:20.638197 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:20.638493 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:20.638827 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 23:12:20.639061 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 23:12:20.640178 21687 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:12:20.727671 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 21564
I20260620 23:12:20.744678 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 21698
Jun 20 23:12:20 dist-test-slave-jkbq krb5kdc[21324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.195@KRBTEST.COM
I20260620 23:12:20.761282 21841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:12:20.745729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39736 (local address 127.10.191.195:41449)
0620 23:12:20.745943 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 23:12:20.745949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:12:20.745971 (+ 22us) server_negotiation.cc:407] Connection header received
0620 23:12:20.746035 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:12:20.746041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:12:20.746120 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:12:20.746234 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:12:20.747522 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:20.748792 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:20.749958 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:12:20.750651 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:12:20.750795 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:12:20.755261 (+ 4466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:12:20.755273 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:12:20.755287 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:12:20.755325 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:12:20.758391 (+ 3066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:20.759268 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:20.759276 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:20.759278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:20.759439 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:12:20.759830 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:12:20.759833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:12:20.759834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:12:20.760179 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:12:20.760406 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 23:12:20.760755 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 23:12:20.760904 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 23:12:20.762010 21821 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:12:20.883906 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 21698
I20260620 23:12:20.899935 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 21337
I20260620 23:12:20.901116 21399 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:12:21.007979 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 21337
2026-06-20T23:12:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4193 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:12:21 dist-test-slave-jkbq krb5kdc[21877](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:12:21 dist-test-slave-jkbq krb5kdc[21877](info): set up 2 sockets
Jun 20 23:12:21 dist-test-slave-jkbq krb5kdc[21877](info): commencing operation
krb5kdc: starting...
W20260620 23:12:22.115872 11007 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Jun 20 23:12:22 dist-test-slave-jkbq krb5kdc[21877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:12:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:12:22Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.10.191.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.10.191.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.10.191.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/krb5kdc/rangeradmin_127.10.191.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.10.191.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/krb5kdc/rangeradmin_127.10.191.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.10.191.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.10.191.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.10.191.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/krb5kdc/rangerlookup_127.10.191.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.10.191.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/krb5kdc/rangerlookup_127.10.191.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.191.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.191.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.191.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/krb5kdc/HTTP_127.10.191.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.191.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/krb5kdc/HTTP_127.10.191.212@KRBTEST.COM.keytab.
I20260620 23:12:22.433843 11007 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskpRJWBp/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/postgres -l logfile start
2026-06-20 23:12:25.010 UTC [21907] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 23:12:25.010 UTC [21907] LOG: listening on IPv4 address "127.10.191.212", port 42031
2026-06-20 23:12:25.013 UTC [21907] LOG: listening on Unix socket "/tmp/.s.PGSQL.42031"
2026-06-20 23:12:25.035 UTC [21910] LOG: database system was shut down at 2026-06-20 23:12:23 UTC
2026-06-20 23:12:25.039 UTC [21907] LOG: database system is ready to accept connections
I20260620 23:12:26.015455 11007 mini_postgres.cc:96] Postgres bound to 42031
2026-06-20 23:12:26.033 UTC [21918] FATAL: database "slave" does not exist
127.10.191.212:42031 - accepting connections
I20260620 23:12:26.034121 11007 mini_ranger.cc:161] Starting Ranger...
I20260620 23:12:26.060876 11007 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 23:12:26.578828 11007 mini_ranger.cc:87] Created ranger Postgres database
I20260620 23:12:26.579043 11007 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ranger-admin
2026-06-20 23:12:26,881 [I] DB FLAVOR :POSTGRES
2026-06-20 23:12:26,882 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 23:12:26,882 [I] Checking connection..
2026-06-20 23:12:26,882 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.191.212:42031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 23:12:27,135 [I] Checking connection passed.
2026-06-20 23:12:27,135 [I] --------- Verifying version history table ---------
2026-06-20 23:12:27,135 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.191.212:42031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 23:12:27,395 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 23:12:27,396 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.191.212:42031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 23:12:27,653 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 23:12:27,654 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 23:12:27,654 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.191.212:42031/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 23:12:27.909 UTC [21982] WARNING: there is no transaction in progress
2026-06-20 23:12:27,916 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 23:12:27,916 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.191.212:42031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 23:12:28,144 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 23:12:28,144 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 23:12:28,144 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.191.212:42031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 23:12:28,384 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.191.212:42031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 23:12:28,614 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.191.212:42031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-jkbq.c.gcp-upstream.internal','N') ;"
2026-06-20 23:12:28,890 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.191.212:42031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 23:12:29,155 [I] Table x_portal_user does not exist in database ranger
2026-06-20 23:12:29,155 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.191.212:42031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 23:12:29,384 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 23:12:29,385 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.191.212:42031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 23:12:29,643 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.191.212:42031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 23:12:29,900 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 23:12:29,900 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.191.212:42031/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 23:12:30.813 UTC [22090] WARNING: there is no transaction in progress
2026-06-20 23:12:30.828 UTC [22090] WARNING: there is no transaction in progress
2026-06-20 23:12:30.844 UTC [22090] WARNING: there is no transaction in progress
2026-06-20 23:12:30.859 UTC [22090] WARNING: there is no transaction in progress
2026-06-20 23:12:30.874 UTC [22090] WARNING: there is no transaction in progress
2026-06-20 23:12:30.893 UTC [22090] WARNING: there is no transaction in progress
2026-06-20 23:12:30.951 UTC [22090] WARNING: there is no transaction in progress
2026-06-20 23:12:30.957 UTC [22090] WARNING: there is no transaction in progress
2026-06-20 23:12:30.971 UTC [22090] WARNING: there is no transaction in progress
2026-06-20 23:12:30.981 UTC [22090] WARNING: there is no transaction in progress
2026-06-20 23:12:30.990 UTC [22090] WARNING: there is no transaction in progress
2026-06-20 23:12:31.002 UTC [22090] WARNING: there is no transaction in progress
2026-06-20 23:12:31.011 UTC [22090] WARNING: there is no transaction in progress
2026-06-20 23:12:31.017 UTC [22090] WARNING: there is no transaction in progress
2026-06-20 23:12:31.024 UTC [22090] WARNING: there is no transaction in progress
2026-06-20 23:12:31.030 UTC [22090] WARNING: there is no transaction in progress
2026-06-20 23:12:31,635 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 23:12:31,635 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.191.212:42031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 23:12:31,888 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 23:12:31,888 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.191.212:42031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 23:12:32,137 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 23:12:32,137 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.191.212:42031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 23:12:32,396 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 23:12:32,396 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.191.212:42031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 23:12:32,621 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.191.212:42031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 23:12:32,847 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.191.212:42031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-jkbq.c.gcp-upstream.internal';"
2026-06-20 23:12:33,094 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 23:12:33,095 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.191.212:42031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 23:12:33,337 [I] DB_PATCHES have already been applied
I20260620 23:12:33.343386 11007 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ranger-admin:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskpRJWBp/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 23:12:33.343505 11007 mini_ranger.cc:193] Using host: 127.10.191.212
I20260620 23:12:33.357941 11007 mini_ranger.cc:227] Ranger admin URL: http://127.10.191.212:33289
Jun 20, 2026 11:12:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 11:12:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 11:12:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 11:12:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 11:12:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 11:12:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 11:12:34 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 23:12:34.493623 11007 mini_ranger.cc:160] Time spent starting Ranger: real 8.460s user 0.002s sys 0.036s
Jun 20, 2026 11:12:54 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 11:12:54 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 20, 2026 11:12:54 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 11:12:54 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 11:12:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 11:12:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 11:12:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 11:12:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 11:12:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 11:12:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 11:12:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 11:12:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 11:12:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 11:12:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 11:12:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 11:12:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 11:12:54 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 11:12:55 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 23:12:56.273819 11007 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.10.191.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.10.191.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.10.191.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/krb5kdc/rangerkms_127.10.191.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.10.191.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/krb5kdc/rangerkms_127.10.191.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.191.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.10.191.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.191.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/krb5kdc/HTTP_127.10.191.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.191.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/krb5kdc/HTTP_127.10.191.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 23:12:56 dist-test-slave-jkbq krb5kdc[21877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997176, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 23:12:56.439144 11007 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 23:12:56.459079 11007 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 23:12:57.289255 11007 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 23:12:57.289440 11007 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ranger-kms
I20260620 23:12:57.289580 11007 mini_ranger_kms.cc:227] Using postgres at 127.10.191.212:42031
2026-06-20 23:12:57,696 [I] DB FLAVOR :POSTGRES
2026-06-20 23:12:57,696 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 23:12:57,696 [I] Checking connection
2026-06-20 23:12:57,696 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.191.212:42031/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 23:12:57,935 [I] connection success
2026-06-20 23:12:57,935 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 23:12:57,936 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 23:12:57,936 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.191.212:42031/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 23:12:58,191 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 23:12:58,191 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 23:12:58,191 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 23:12:58,192 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.191.212:42031/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 23:12:58,449 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 23:12:58.887943 11007 mini_ranger_kms.cc:310] Created kms service
I20260620 23:12:59.081566 11007 mini_ranger_kms.cc:326] Created kudu user
I20260620 23:12:59.125305 11007 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 23:12:59 dist-test-slave-jkbq krb5kdc[21877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997176, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.10.191.212@KRBTEST.COM
I20260620 23:12:59.493482 11007 mini_ranger_kms.cc:384] Added ranger policy
I20260620 23:12:59.493801 11007 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ranger-kms:/tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 23:12:59.493870 11007 mini_ranger_kms.cc:240] Using host: 127.10.191.212
I20260620 23:12:59.514812 11007 mini_ranger_kms.cc:276] Ranger KMS PID: 22270
I20260620 23:12:59.515036 11007 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.10.191.212:35297
Jun 20, 2026 11:13:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 11:13:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 11:13:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 11:13:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 11:13:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 11:13:01 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 23:13:01.854990 11007 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.416s user 0.136s sys 0.285s
I20260620 23:13:01.855268 11007 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 23:13:01.855480 11007 mini_ranger_kms.cc:401] 127.10.191.212:35297/kms/v1/keys
Jun 20 23:13:01 dist-test-slave-jkbq krb5kdc[21877](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781997181, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.191.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 23:13:02 dist-test-slave-jkbq krb5kdc[21877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997181, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.191.212@KRBTEST.COM for HTTP/127.10.191.212@KRBTEST.COM
Jun 20 23:13:02 dist-test-slave-jkbq krb5kdc[21877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997181, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.191.212@KRBTEST.COM for HTTP/127.10.191.212@KRBTEST.COM
Jun 20 23:13:04 dist-test-slave-jkbq krb5kdc[21877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.10.191.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
I20260620 23:13:04.904007 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:39701
--webserver_interface=127.10.191.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:43963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:39701
--ranger_config_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.191.212:35297/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:13:05.224272 22321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:05.224622 22321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:05.224772 22321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:05.231989 22321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:13:05.232138 22321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:05.232240 22321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:05.232347 22321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:13:05.232425 22321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:13:05.242411 22321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:43963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/master-0/wal
--ranger_kms_url=127.10.191.212:35297/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:39701
--ranger_config_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:39701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:13:05.243765 22321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:05.245041 22321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:05.258286 22327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:05.258312 22326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:05.258925 22329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:05.259661 22321 server_base.cc:1034] running on GCE node
I20260620 23:13:05.260569 22321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:05.262254 22321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:05.263484 22321 hybrid_clock.cc:648] HybridClock initialized: now 1781997185263422 us; error 65 us; skew 500 ppm
I20260620 23:13:05.263865 22321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:13:05 dist-test-slave-jkbq krb5kdc[21877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997185, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:05.270797 22321 init.cc:377] Logged in from keytab as kudu/127.10.191.254@KRBTEST.COM (short username kudu)
I20260620 23:13:05.272415 22321 webserver.cc:460] Webserver started at http://127.10.191.254:32869/ using document root <none> and password file <none>
I20260620 23:13:05.273092 22321 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:05.273216 22321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:05.273538 22321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:13:05 dist-test-slave-jkbq krb5kdc[21877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997185, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for HTTP/127.10.191.212@KRBTEST.COM
I20260620 23:13:05.321710 22321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "1d50e5942de04f3588a22df345770d4d"
format_stamp: "Formatted at 2026-06-20 23:13:05 on dist-test-slave-jkbq"
server_key: "9e65f0426a08ea0655a93d29ac89759e"
server_key_iv: "9d6b534ce2437ed1b8850073b6d96a01"
server_key_version: "kuduclusterkey@0"
I20260620 23:13:05.322564 22321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "1d50e5942de04f3588a22df345770d4d"
format_stamp: "Formatted at 2026-06-20 23:13:05 on dist-test-slave-jkbq"
server_key: "9e65f0426a08ea0655a93d29ac89759e"
server_key_iv: "9d6b534ce2437ed1b8850073b6d96a01"
server_key_version: "kuduclusterkey@0"
I20260620 23:13:05.328110 22321 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 23:13:05.344024 22336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:05.345635 22321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:13:05.345836 22321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "1d50e5942de04f3588a22df345770d4d"
format_stamp: "Formatted at 2026-06-20 23:13:05 on dist-test-slave-jkbq"
server_key: "9e65f0426a08ea0655a93d29ac89759e"
server_key_iv: "9d6b534ce2437ed1b8850073b6d96a01"
server_key_version: "kuduclusterkey@0"
I20260620 23:13:05.346068 22321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:05.370193 22321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:05.374329 22321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:05.374634 22321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:05.394136 22321 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:39701
I20260620 23:13:05.394170 22387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:39701 every 8 connection(s)
I20260620 23:13:05.395745 22321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:13:05.399565 22388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:05.401937 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 22321
I20260620 23:13:05.402179 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:13:05.402840 11007 external_mini_cluster.cc:1428] Setting key b44fda684022c02c7f83170386a35fb4
I20260620 23:13:05.411432 22388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d50e5942de04f3588a22df345770d4d: Bootstrap starting.
I20260620 23:13:05.415231 22388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d50e5942de04f3588a22df345770d4d: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:05.416527 22388 log.cc:824] T 00000000000000000000000000000000 P 1d50e5942de04f3588a22df345770d4d: Log is configured to *not* fsync() on all Append() calls
Jun 20 23:13:05 dist-test-slave-jkbq krb5kdc[21877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:13:05.419847 22388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d50e5942de04f3588a22df345770d4d: No bootstrap required, opened a new log
I20260620 23:13:05.426295 22391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:05.404860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39520 (local address 127.10.191.254:39701)
0620 23:13:05.405642 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:05.405653 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:05.405694 (+ 41us) server_negotiation.cc:407] Connection header received
0620 23:13:05.407458 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:05.407480 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:05.407869 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:05.408202 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:05.409802 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:05.411172 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:05.412952 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:05.413627 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:05.413758 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:05.418137 (+ 4379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:05.418148 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:05.418164 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:05.418202 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:05.422092 (+ 3890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:05.422767 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:05.422781 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:05.422789 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:05.422998 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:05.423348 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:05.423353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:05.423357 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:05.424024 (+ 667us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:05.424324 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:05.424592 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 23:13:05.424972 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":387,"threads_started":1}
I20260620 23:13:05.429338 22388 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d50e5942de04f3588a22df345770d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d50e5942de04f3588a22df345770d4d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39701 } }
I20260620 23:13:05.429641 22388 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d50e5942de04f3588a22df345770d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:05.429713 22388 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d50e5942de04f3588a22df345770d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d50e5942de04f3588a22df345770d4d, State: Initialized, Role: FOLLOWER
I20260620 23:13:05.430166 22388 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d50e5942de04f3588a22df345770d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d50e5942de04f3588a22df345770d4d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39701 } }
I20260620 23:13:05.430358 22388 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d50e5942de04f3588a22df345770d4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:13:05.430449 22388 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d50e5942de04f3588a22df345770d4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:13:05.430567 22388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d50e5942de04f3588a22df345770d4d [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:05.432619 22388 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d50e5942de04f3588a22df345770d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d50e5942de04f3588a22df345770d4d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39701 } }
I20260620 23:13:05.433048 22388 leader_election.cc:302] T 00000000000000000000000000000000 P 1d50e5942de04f3588a22df345770d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d50e5942de04f3588a22df345770d4d; no voters:
I20260620 23:13:05.433552 22388 leader_election.cc:288] T 00000000000000000000000000000000 P 1d50e5942de04f3588a22df345770d4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:13:05.433715 22393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d50e5942de04f3588a22df345770d4d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:13:05.434370 22393 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d50e5942de04f3588a22df345770d4d [term 1 LEADER]: Becoming Leader. State: Replica: 1d50e5942de04f3588a22df345770d4d, State: Running, Role: LEADER
I20260620 23:13:05.434999 22393 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d50e5942de04f3588a22df345770d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d50e5942de04f3588a22df345770d4d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39701 } }
I20260620 23:13:05.435662 22388 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d50e5942de04f3588a22df345770d4d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:13:05.437882 22395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d50e5942de04f3588a22df345770d4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d50e5942de04f3588a22df345770d4d. Latest consensus state: current_term: 1 leader_uuid: "1d50e5942de04f3588a22df345770d4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d50e5942de04f3588a22df345770d4d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39701 } } }
I20260620 23:13:05.438256 22395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d50e5942de04f3588a22df345770d4d [sys.catalog]: This master's current role is: LEADER
I20260620 23:13:05.439779 22394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d50e5942de04f3588a22df345770d4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d50e5942de04f3588a22df345770d4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d50e5942de04f3588a22df345770d4d" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39701 } } }
I20260620 23:13:05.440028 22394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d50e5942de04f3588a22df345770d4d [sys.catalog]: This master's current role is: LEADER
I20260620 23:13:05.440510 22401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:13:05.445622 22401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:13:05.448272 22388 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 23:13:05.463088 22401 catalog_manager.cc:1347] Generated new cluster ID: 49fbffce734f437f808292dc11c16b7b
I20260620 23:13:05.463227 22401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:13:05.481287 22401 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:13:05.482604 22401 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:13:05.489941 22401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d50e5942de04f3588a22df345770d4d: Generated new TSK 0
I20260620 23:13:05.490741 22401 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.191.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab.
Entry for principal kudu/127.10.191.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab.
I20260620 23:13:06.242235 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.191.212:35297/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.193:0
--local_ip_for_outbound_sockets=127.10.191.193
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:39701
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:43963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:13:06 dist-test-slave-jkbq krb5kdc[21877](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781997186, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 23:13:06.648550 22433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:06.648867 22433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:06.649048 22433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:06.657832 22433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:06.658033 22433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:06.658259 22433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.193
I20260620 23:13:06.669644 22433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:43963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.10.191.212:35297/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:39701
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.193
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:13:06.671111 22433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:06.672423 22433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:06.691084 22443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:06.691066 22444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:06.698380 22446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:06.699309 22433 server_base.cc:1034] running on GCE node
I20260620 23:13:06.699982 22433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:06.707142 22433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:06.710551 22433 hybrid_clock.cc:648] HybridClock initialized: now 1781997186710475 us; error 65 us; skew 500 ppm
I20260620 23:13:06.710981 22433 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:13:06 dist-test-slave-jkbq krb5kdc[21877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997186, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:06.714421 22433 init.cc:377] Logged in from keytab as kudu/127.10.191.193@KRBTEST.COM (short username kudu)
I20260620 23:13:06.722165 22433 webserver.cc:460] Webserver started at http://127.10.191.193:46075/ using document root <none> and password file <none>
I20260620 23:13:06.722975 22433 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:06.723121 22433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:06.723454 22433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:13:06 dist-test-slave-jkbq krb5kdc[21877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997186, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for HTTP/127.10.191.212@KRBTEST.COM
I20260620 23:13:06.743784 22433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-0/data/instance:
uuid: "3430624656cf4ddeb6381b272ae146e0"
format_stamp: "Formatted at 2026-06-20 23:13:06 on dist-test-slave-jkbq"
server_key: "24703de71d15247e709b49eda07a1397"
server_key_iv: "76f22055d14692f9e617ea51fe1eef13"
server_key_version: "kuduclusterkey@0"
I20260620 23:13:06.744977 22433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance:
uuid: "3430624656cf4ddeb6381b272ae146e0"
format_stamp: "Formatted at 2026-06-20 23:13:06 on dist-test-slave-jkbq"
server_key: "24703de71d15247e709b49eda07a1397"
server_key_iv: "76f22055d14692f9e617ea51fe1eef13"
server_key_version: "kuduclusterkey@0"
I20260620 23:13:06.760421 22433 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.008s sys 0.004s
I20260620 23:13:06.799510 22453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:06.804684 22433 fs_manager.cc:725] Time spent opening block manager: real 0.026s user 0.003s sys 0.011s
I20260620 23:13:06.805011 22433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-0/wal
uuid: "3430624656cf4ddeb6381b272ae146e0"
format_stamp: "Formatted at 2026-06-20 23:13:06 on dist-test-slave-jkbq"
server_key: "24703de71d15247e709b49eda07a1397"
server_key_iv: "76f22055d14692f9e617ea51fe1eef13"
server_key_version: "kuduclusterkey@0"
I20260620 23:13:06.805382 22433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:06.868623 22433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:06.875399 22433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:06.875804 22433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:06.877051 22433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:13:06.878825 22433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:06.878937 22433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 23:13:06.879070 22433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:06.879133 22433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:06.950719 22433 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.193:40663
I20260620 23:13:06.950780 22565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.193:40663 every 8 connection(s)
I20260620 23:13:06.952553 22433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
I20260620 23:13:06.957937 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 22433
I20260620 23:13:06.958165 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance
I20260620 23:13:06.959095 11007 external_mini_cluster.cc:1428] Setting key 0e5a17cd373f0e545ab163c78a5039bd
Jun 20 23:13:06 dist-test-slave-jkbq krb5kdc[21877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997186, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.191.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.194@KRBTEST.COM" created.
I20260620 23:13:07.008040 22568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:06.956955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39701 (local address 127.10.191.193:37271)
0620 23:13:06.976281 (+ 19326us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:06.976371 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:06.979959 (+ 3588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:06.986254 (+ 6295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:06.986271 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:06.987048 (+ 777us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:06.987943 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:06.987972 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:06.989720 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:06.989728 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:06.992109 (+ 2381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:06.992140 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:06.993654 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:06.993663 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:06.994045 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:06.994959 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:06.995008 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:06.998157 (+ 3149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:07.002739 (+ 4582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:07.002748 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:07.002768 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:07.003387 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:07.003843 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:07.003850 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:07.003855 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:07.004131 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:07.004896 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:07.004904 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:07.005415 (+ 511us) client_negotiation.cc:769] Sending connection context
0620 23:13:07.005741 (+ 326us) client_negotiation.cc:240] Negotiation successful
0620 23:13:07.006393 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18834,"thread_start_us":721,"threads_started":1}
I20260620 23:13:07.010092 22569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:06.957575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:37271 (local address 127.10.191.254:39701)
0620 23:13:06.985308 (+ 27733us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:06.985315 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:06.985343 (+ 28us) server_negotiation.cc:407] Connection header received
0620 23:13:06.985417 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:06.985423 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:06.985526 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:06.985659 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:06.988195 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:06.989458 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:06.992320 (+ 2862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:06.993160 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:06.993296 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:06.998480 (+ 5184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:06.998487 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:06.998495 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:06.998548 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:07.002518 (+ 3970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:07.003536 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:07.003541 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:07.003545 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:07.003701 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:07.004274 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:07.004278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:07.004281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:07.004643 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:07.004816 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:07.009319 (+ 4503us) server_negotiation.cc:299] Negotiation successful
0620 23:13:07.009602 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27579,"thread_start_us":27448,"threads_started":1}
I20260620 23:13:07.011984 22566 heartbeater.cc:347] Connected to a master server at 127.10.191.254:39701
I20260620 23:13:07.012336 22566 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:07.013247 22566 heartbeater.cc:511] Master 127.10.191.254:39701 requested a full tablet report, sending...
I20260620 23:13:07.015707 22353 ts_manager.cc:194] Registered new tserver with Master: 3430624656cf4ddeb6381b272ae146e0 (127.10.191.193:40663)
I20260620 23:13:07.017841 22353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.193@KRBTEST.COM'} at 127.10.191.193:37271
Jun 20 23:13:07 dist-test-slave-jkbq krb5kdc[21877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997186, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for HTTP/127.10.191.212@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab.
Entry for principal kudu/127.10.191.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab.
I20260620 23:13:07.052332 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.191.212:35297/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.194:0
--local_ip_for_outbound_sockets=127.10.191.194
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:39701
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:43963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:13:07 dist-test-slave-jkbq krb5kdc[21877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997186, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for HTTP/127.10.191.212@KRBTEST.COM
W20260620 23:13:07.552450 22572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:07.552938 22572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:07.553145 22572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:07.564836 22572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:07.565050 22572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:07.565295 22572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.194
I20260620 23:13:07.580915 22572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:43963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.10.191.212:35297/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:39701
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.194
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:13:07.583204 22572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:07.585018 22572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:07.618662 22580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:07.626143 22583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:07.626547 22572 server_base.cc:1034] running on GCE node
W20260620 23:13:07.631230 22579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:07.632218 22572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:07.641074 22572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:07.643253 22572 hybrid_clock.cc:648] HybridClock initialized: now 1781997187643035 us; error 131 us; skew 500 ppm
I20260620 23:13:07.643720 22572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:13:07 dist-test-slave-jkbq krb5kdc[21877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997187, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:07.649755 22572 init.cc:377] Logged in from keytab as kudu/127.10.191.194@KRBTEST.COM (short username kudu)
I20260620 23:13:07.674551 22572 webserver.cc:460] Webserver started at http://127.10.191.194:33515/ using document root <none> and password file <none>
I20260620 23:13:07.675618 22572 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:07.675853 22572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:07.676313 22572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:13:07 dist-test-slave-jkbq krb5kdc[21877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997187, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for HTTP/127.10.191.212@KRBTEST.COM
I20260620 23:13:07.713490 22572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-1/data/instance:
uuid: "c390418022c145ac906c82aba8df0c17"
format_stamp: "Formatted at 2026-06-20 23:13:07 on dist-test-slave-jkbq"
server_key: "4f1751402796d1e25f75627b05bc39f5"
server_key_iv: "f5486b9f4dbf85a6003f212725bfd093"
server_key_version: "kuduclusterkey@0"
I20260620 23:13:07.719515 22572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance:
uuid: "c390418022c145ac906c82aba8df0c17"
format_stamp: "Formatted at 2026-06-20 23:13:07 on dist-test-slave-jkbq"
server_key: "4f1751402796d1e25f75627b05bc39f5"
server_key_iv: "f5486b9f4dbf85a6003f212725bfd093"
server_key_version: "kuduclusterkey@0"
I20260620 23:13:07.732882 22572 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.005s sys 0.000s
I20260620 23:13:07.775256 22591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:07.778247 22572 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.005s sys 0.001s
I20260620 23:13:07.778499 22572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-1/wal
uuid: "c390418022c145ac906c82aba8df0c17"
format_stamp: "Formatted at 2026-06-20 23:13:07 on dist-test-slave-jkbq"
server_key: "4f1751402796d1e25f75627b05bc39f5"
server_key_iv: "f5486b9f4dbf85a6003f212725bfd093"
server_key_version: "kuduclusterkey@0"
I20260620 23:13:07.778852 22572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:07.818795 22572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:07.823978 22572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:07.824307 22572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:07.825510 22572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:13:07.827211 22572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:07.827306 22572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:07.827438 22572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:07.827504 22572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:07.866786 22572 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.194:36717
I20260620 23:13:07.866803 22708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.194:36717 every 8 connection(s)
I20260620 23:13:07.868284 22572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
I20260620 23:13:07.868685 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 22572
I20260620 23:13:07.868868 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance
I20260620 23:13:07.869453 11007 external_mini_cluster.cc:1428] Setting key 653d7b6a0dbcfbc8755f48512f9613df
Jun 20 23:13:07 dist-test-slave-jkbq krb5kdc[21877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997187, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:13:07.894348 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:07.871088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:35573 (local address 127.10.191.254:39701)
0620 23:13:07.876997 (+ 5909us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:07.877001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:07.877015 (+ 14us) server_negotiation.cc:407] Connection header received
0620 23:13:07.877059 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:07.877063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:07.877133 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:07.877258 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:07.878938 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:07.879662 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:07.881689 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:07.882317 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:07.882470 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:07.887029 (+ 4559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:07.887036 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:07.887041 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:07.887078 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:07.890618 (+ 3540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:07.891576 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:07.891582 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:07.891585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:07.891736 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:07.892270 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:07.892275 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:07.892278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:07.892615 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:07.892757 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:07.893734 (+ 977us) server_negotiation.cc:299] Negotiation successful
0620 23:13:07.893966 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5799,"thread_start_us":5693,"threads_started":1}
I20260620 23:13:07.895638 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:07.871882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39701 (local address 127.10.191.194:35573)
0620 23:13:07.872846 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:07.872906 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:07.876121 (+ 3215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:07.877535 (+ 1414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:07.877547 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:07.878172 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:07.878736 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:07.878754 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:07.879816 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:07.879823 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:07.881502 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:07.881533 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:07.882830 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:07.882840 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:07.883181 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:07.883993 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:07.884031 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:07.886763 (+ 2732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:07.890772 (+ 4009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:07.890783 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:07.890802 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:07.891446 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:07.891865 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:07.891871 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:07.891874 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:07.892127 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:07.892763 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:07.892773 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:07.893255 (+ 482us) client_negotiation.cc:769] Sending connection context
0620 23:13:07.893594 (+ 339us) client_negotiation.cc:240] Negotiation successful
0620 23:13:07.894144 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":5384,"threads_started":1}
WARNING: no policy specified for kudu/127.10.191.195@KRBTEST.COM; defaulting to no policy
I20260620 23:13:07.897176 22709 heartbeater.cc:347] Connected to a master server at 127.10.191.254:39701
I20260620 23:13:07.897554 22709 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:07.898260 22709 heartbeater.cc:511] Master 127.10.191.254:39701 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.195@KRBTEST.COM" created.
I20260620 23:13:07.899997 22353 ts_manager.cc:194] Registered new tserver with Master: c390418022c145ac906c82aba8df0c17 (127.10.191.194:36717)
I20260620 23:13:07.901053 22353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.194@KRBTEST.COM'} at 127.10.191.194:35573
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab.
Entry for principal kudu/127.10.191.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab.
I20260620 23:13:07.926239 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.191.212:35297/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.195:0
--local_ip_for_outbound_sockets=127.10.191.195
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:39701
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:43963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:13:08.024070 22566 heartbeater.cc:503] Master 127.10.191.254:39701 was elected leader, sending a full tablet report...
W20260620 23:13:08.247664 22715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:08.247927 22715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:08.248081 22715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:08.255318 22715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:08.255425 22715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:08.255595 22715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.195
I20260620 23:13:08.265200 22715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:43963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.10.191.212:35297/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:39701
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.195
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:13:08.266455 22715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:08.267827 22715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:08.280897 22723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:08.280933 22720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:08.284711 22715 server_base.cc:1034] running on GCE node
W20260620 23:13:08.286060 22721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:08.286928 22715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:08.288398 22715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:08.289714 22715 hybrid_clock.cc:648] HybridClock initialized: now 1781997188289639 us; error 63 us; skew 500 ppm
I20260620 23:13:08.290079 22715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:13:08 dist-test-slave-jkbq krb5kdc[21877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997188, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:08.294823 22715 init.cc:377] Logged in from keytab as kudu/127.10.191.195@KRBTEST.COM (short username kudu)
I20260620 23:13:08.297243 22715 webserver.cc:460] Webserver started at http://127.10.191.195:38515/ using document root <none> and password file <none>
I20260620 23:13:08.298331 22715 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:08.298460 22715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:08.298825 22715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:13:08 dist-test-slave-jkbq krb5kdc[21877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997188, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for HTTP/127.10.191.212@KRBTEST.COM
I20260620 23:13:08.318492 22715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-2/data/instance:
uuid: "6305bba89eae4eeb9a2e37abb05c8cf6"
format_stamp: "Formatted at 2026-06-20 23:13:08 on dist-test-slave-jkbq"
server_key: "c1e82be1b04b5ed1b4a550a5ff8da415"
server_key_iv: "62bd63170ce10ec4acc43e7f9e9f23a8"
server_key_version: "kuduclusterkey@0"
I20260620 23:13:08.319314 22715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance:
uuid: "6305bba89eae4eeb9a2e37abb05c8cf6"
format_stamp: "Formatted at 2026-06-20 23:13:08 on dist-test-slave-jkbq"
server_key: "c1e82be1b04b5ed1b4a550a5ff8da415"
server_key_iv: "62bd63170ce10ec4acc43e7f9e9f23a8"
server_key_version: "kuduclusterkey@0"
I20260620 23:13:08.324818 22715 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 23:13:08.342283 22730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:08.343918 22715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:13:08.344105 22715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-2/wal
uuid: "6305bba89eae4eeb9a2e37abb05c8cf6"
format_stamp: "Formatted at 2026-06-20 23:13:08 on dist-test-slave-jkbq"
server_key: "c1e82be1b04b5ed1b4a550a5ff8da415"
server_key_iv: "62bd63170ce10ec4acc43e7f9e9f23a8"
server_key_version: "kuduclusterkey@0"
I20260620 23:13:08.344332 22715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:08.364099 22715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:08.368187 22715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:08.368485 22715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:08.369415 22715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:13:08.370847 22715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:08.371003 22715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260620 23:13:08.371155 22715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:08.371239 22715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:08.417817 22715 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.195:35263
I20260620 23:13:08.417838 22842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.195:35263 every 8 connection(s)
I20260620 23:13:08.419142 22715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
I20260620 23:13:08.427910 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 22715
I20260620 23:13:08.428146 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance
I20260620 23:13:08.428720 11007 external_mini_cluster.cc:1428] Setting key ebc201cb9a6174fb9e8f7a8fd5a78e3f
Jun 20 23:13:08 dist-test-slave-jkbq krb5kdc[21877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997188, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:13:08.441665 22845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:08.421877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:36345 (local address 127.10.191.254:39701)
0620 23:13:08.422326 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:08.422332 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:08.423496 (+ 1164us) server_negotiation.cc:407] Connection header received
0620 23:13:08.425928 (+ 2432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:08.425932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:08.426015 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:08.426109 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:08.428040 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.429146 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.431330 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.431784 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.431918 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:08.435667 (+ 3749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:08.435672 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:08.435675 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:08.435705 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:08.438471 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:08.439362 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:08.439366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:08.439368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:08.439490 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:08.439974 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:08.439977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:08.439978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:08.440223 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:08.440326 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:08.441197 (+ 871us) server_negotiation.cc:299] Negotiation successful
0620 23:13:08.441370 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":214,"threads_started":1}
I20260620 23:13:08.443030 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:08.422496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39701 (local address 127.10.191.195:36345)
0620 23:13:08.423296 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:08.423355 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:08.425639 (+ 2284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:08.426402 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:08.426413 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:08.427046 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:08.427822 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:08.427849 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.429305 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.429311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:08.431178 (+ 1867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:08.431197 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.432030 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.432037 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:08.432273 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:08.432957 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:08.432980 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:08.435353 (+ 2373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:08.438633 (+ 3280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:08.438641 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:08.438658 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:08.439248 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:08.439601 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:08.439606 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:08.439610 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:08.439862 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:08.440345 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:08.440353 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:08.440792 (+ 439us) client_negotiation.cc:769] Sending connection context
0620 23:13:08.441072 (+ 280us) client_negotiation.cc:240] Negotiation successful
0620 23:13:08.441573 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":321,"threads_started":1}
I20260620 23:13:08.444888 22843 heartbeater.cc:347] Connected to a master server at 127.10.191.254:39701
I20260620 23:13:08.445210 22843 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:08.446054 22843 heartbeater.cc:511] Master 127.10.191.254:39701 requested a full tablet report, sending...
I20260620 23:13:08.447652 22352 ts_manager.cc:194] Registered new tserver with Master: 6305bba89eae4eeb9a2e37abb05c8cf6 (127.10.191.195:35263)
I20260620 23:13:08.448618 22352 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.195@KRBTEST.COM'} at 127.10.191.195:36345
I20260620 23:13:08.456887 11007 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:13:08.476882 22845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:08.462546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39522 (local address 127.10.191.254:39701)
0620 23:13:08.462774 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:08.462784 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:08.463322 (+ 538us) server_negotiation.cc:407] Connection header received
0620 23:13:08.463518 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:08.463522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:08.463599 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:08.463681 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:08.465462 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.466172 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.468247 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.468674 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.468769 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:08.471583 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:08.471588 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:08.471591 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:08.471621 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:08.474121 (+ 2500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:08.474950 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:08.474958 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:08.474962 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:08.475103 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:08.475530 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:08.475535 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:08.475540 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:08.475788 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:08.476029 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:08.476355 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 23:13:08.476581 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 23:13:08.480845 22352 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:13:08.488478 22352 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:13:08.526854 22860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:08.501295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48436 (local address 127.10.191.193:40663)
0620 23:13:08.517149 (+ 15854us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:08.517162 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:08.518295 (+ 1133us) server_negotiation.cc:407] Connection header received
0620 23:13:08.518393 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:08.518401 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:08.518623 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:08.518806 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:13:08.519811 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.521341 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.524350 (+ 3009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.525653 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.525847 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:08.526033 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:08.526349 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 23:13:08.526509 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15650,"thread_start_us":15485,"threads_started":1}
I20260620 23:13:08.526921 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:08.500612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:40663 (local address 127.0.0.1:48436)
0620 23:13:08.517949 (+ 17337us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:08.517969 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:08.518104 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:08.519084 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:08.519094 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:08.519117 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:13:08.519600 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:08.519615 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.521491 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.521497 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:08.524152 (+ 2655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:08.524164 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.525857 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.525863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:08.526080 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:08.526160 (+ 80us) client_negotiation.cc:769] Sending connection context
0620 23:13:08.526290 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 23:13:08.526497 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17217,"mutex_wait_us":165,"thread_start_us":17088,"threads_started":1}
I20260620 23:13:08.527395 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:08.501082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.194:36717 (local address 127.0.0.1:40606)
0620 23:13:08.519822 (+ 18740us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:08.519840 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:08.519976 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:08.520449 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:08.520455 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:08.520483 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:13:08.520869 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:08.520887 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.522604 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.522610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:08.524248 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:08.524267 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.525936 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.525943 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:08.526159 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:08.526201 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 23:13:08.526289 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 23:13:08.526482 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18620,"mutex_wait_us":69,"thread_start_us":18464,"threads_started":1}
I20260620 23:13:08.527872 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:08.501695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40606 (local address 127.10.191.194:36717)
0620 23:13:08.519325 (+ 17630us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:08.519337 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:08.519948 (+ 611us) server_negotiation.cc:407] Connection header received
0620 23:13:08.520120 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:08.520125 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:08.520268 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:08.520476 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:13:08.521083 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.522455 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.524429 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.525686 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.525899 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:08.526073 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:08.526659 (+ 586us) server_negotiation.cc:299] Negotiation successful
0620 23:13:08.526814 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17482,"thread_start_us":17381,"threads_started":1}
I20260620 23:13:08.528816 22856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:08.500301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.195:35263 (local address 127.0.0.1:50276)
0620 23:13:08.515472 (+ 15171us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:08.515522 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:08.515748 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:08.518649 (+ 2901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:08.518660 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:08.518690 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:13:08.519114 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:08.519126 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.521160 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.521166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:08.524151 (+ 2985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:08.524164 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.528058 (+ 3894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.528066 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:08.528206 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:08.528227 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 23:13:08.528324 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 23:13:08.528454 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14837,"mutex_wait_us":121,"thread_start_us":15629,"threads_started":1}
I20260620 23:13:08.528862 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:08.500628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50276 (local address 127.10.191.195:35263)
0620 23:13:08.517689 (+ 17061us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:08.517699 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:08.517737 (+ 38us) server_negotiation.cc:407] Connection header received
0620 23:13:08.517839 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:08.517847 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:08.518124 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:08.518304 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:13:08.519368 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.521020 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.526121 (+ 5101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.527590 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.527759 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:08.527914 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:08.528386 (+ 472us) server_negotiation.cc:299] Negotiation successful
0620 23:13:08.528543 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16889,"thread_start_us":16721,"threads_started":1}
I20260620 23:13:08.530787 22501 tablet_service.cc:1467] Processing CreateTablet for tablet d61971a67ddd4c4d93d28ec056b2a852 (DEFAULT_TABLE table=test-table [id=4d466850b1714bf5a2ff189b8e06ab36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:13:08.531632 22644 tablet_service.cc:1467] Processing CreateTablet for tablet d61971a67ddd4c4d93d28ec056b2a852 (DEFAULT_TABLE table=test-table [id=4d466850b1714bf5a2ff189b8e06ab36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:13:08.532030 22501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d61971a67ddd4c4d93d28ec056b2a852. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:08.532558 22644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d61971a67ddd4c4d93d28ec056b2a852. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:08.537112 22778 tablet_service.cc:1467] Processing CreateTablet for tablet d61971a67ddd4c4d93d28ec056b2a852 (DEFAULT_TABLE table=test-table [id=4d466850b1714bf5a2ff189b8e06ab36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:13:08.538446 22778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d61971a67ddd4c4d93d28ec056b2a852. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:08.543260 22862 tablet_bootstrap.cc:492] T d61971a67ddd4c4d93d28ec056b2a852 P 3430624656cf4ddeb6381b272ae146e0: Bootstrap starting.
I20260620 23:13:08.546128 22863 tablet_bootstrap.cc:492] T d61971a67ddd4c4d93d28ec056b2a852 P c390418022c145ac906c82aba8df0c17: Bootstrap starting.
I20260620 23:13:08.547052 22862 tablet_bootstrap.cc:654] T d61971a67ddd4c4d93d28ec056b2a852 P 3430624656cf4ddeb6381b272ae146e0: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:08.548532 22862 log.cc:824] T d61971a67ddd4c4d93d28ec056b2a852 P 3430624656cf4ddeb6381b272ae146e0: Log is configured to *not* fsync() on all Append() calls
I20260620 23:13:08.549649 22863 tablet_bootstrap.cc:654] T d61971a67ddd4c4d93d28ec056b2a852 P c390418022c145ac906c82aba8df0c17: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:08.550926 22863 log.cc:824] T d61971a67ddd4c4d93d28ec056b2a852 P c390418022c145ac906c82aba8df0c17: Log is configured to *not* fsync() on all Append() calls
I20260620 23:13:08.551891 22862 tablet_bootstrap.cc:492] T d61971a67ddd4c4d93d28ec056b2a852 P 3430624656cf4ddeb6381b272ae146e0: No bootstrap required, opened a new log
I20260620 23:13:08.552162 22862 ts_tablet_manager.cc:1397] T d61971a67ddd4c4d93d28ec056b2a852 P 3430624656cf4ddeb6381b272ae146e0: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260620 23:13:08.553972 22863 tablet_bootstrap.cc:492] T d61971a67ddd4c4d93d28ec056b2a852 P c390418022c145ac906c82aba8df0c17: No bootstrap required, opened a new log
I20260620 23:13:08.554217 22863 ts_tablet_manager.cc:1397] T d61971a67ddd4c4d93d28ec056b2a852 P c390418022c145ac906c82aba8df0c17: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 23:13:08.559535 22866 tablet_bootstrap.cc:492] T d61971a67ddd4c4d93d28ec056b2a852 P 6305bba89eae4eeb9a2e37abb05c8cf6: Bootstrap starting.
I20260620 23:13:08.562618 22866 tablet_bootstrap.cc:654] T d61971a67ddd4c4d93d28ec056b2a852 P 6305bba89eae4eeb9a2e37abb05c8cf6: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:08.562978 22862 raft_consensus.cc:348] T d61971a67ddd4c4d93d28ec056b2a852 P 3430624656cf4ddeb6381b272ae146e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6305bba89eae4eeb9a2e37abb05c8cf6" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 35263 } } peers { permanent_uuid: "3430624656cf4ddeb6381b272ae146e0" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40663 } } peers { permanent_uuid: "c390418022c145ac906c82aba8df0c17" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 36717 } }
I20260620 23:13:08.563431 22862 raft_consensus.cc:374] T d61971a67ddd4c4d93d28ec056b2a852 P 3430624656cf4ddeb6381b272ae146e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:08.563529 22862 raft_consensus.cc:729] T d61971a67ddd4c4d93d28ec056b2a852 P 3430624656cf4ddeb6381b272ae146e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3430624656cf4ddeb6381b272ae146e0, State: Initialized, Role: FOLLOWER
I20260620 23:13:08.563872 22866 log.cc:824] T d61971a67ddd4c4d93d28ec056b2a852 P 6305bba89eae4eeb9a2e37abb05c8cf6: Log is configured to *not* fsync() on all Append() calls
I20260620 23:13:08.564154 22862 consensus_queue.cc:260] T d61971a67ddd4c4d93d28ec056b2a852 P 3430624656cf4ddeb6381b272ae146e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6305bba89eae4eeb9a2e37abb05c8cf6" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 35263 } } peers { permanent_uuid: "3430624656cf4ddeb6381b272ae146e0" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40663 } } peers { permanent_uuid: "c390418022c145ac906c82aba8df0c17" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 36717 } }
I20260620 23:13:08.564752 22863 raft_consensus.cc:348] T d61971a67ddd4c4d93d28ec056b2a852 P c390418022c145ac906c82aba8df0c17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6305bba89eae4eeb9a2e37abb05c8cf6" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 35263 } } peers { permanent_uuid: "3430624656cf4ddeb6381b272ae146e0" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40663 } } peers { permanent_uuid: "c390418022c145ac906c82aba8df0c17" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 36717 } }
I20260620 23:13:08.565270 22863 raft_consensus.cc:374] T d61971a67ddd4c4d93d28ec056b2a852 P c390418022c145ac906c82aba8df0c17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:08.565400 22863 raft_consensus.cc:729] T d61971a67ddd4c4d93d28ec056b2a852 P c390418022c145ac906c82aba8df0c17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c390418022c145ac906c82aba8df0c17, State: Initialized, Role: FOLLOWER
I20260620 23:13:08.566121 22863 consensus_queue.cc:260] T d61971a67ddd4c4d93d28ec056b2a852 P c390418022c145ac906c82aba8df0c17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6305bba89eae4eeb9a2e37abb05c8cf6" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 35263 } } peers { permanent_uuid: "3430624656cf4ddeb6381b272ae146e0" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40663 } } peers { permanent_uuid: "c390418022c145ac906c82aba8df0c17" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 36717 } }
I20260620 23:13:08.570683 22709 heartbeater.cc:503] Master 127.10.191.254:39701 was elected leader, sending a full tablet report...
I20260620 23:13:08.571751 22863 ts_tablet_manager.cc:1428] T d61971a67ddd4c4d93d28ec056b2a852 P c390418022c145ac906c82aba8df0c17: Time spent starting tablet: real 0.017s user 0.010s sys 0.008s
I20260620 23:13:08.572263 22862 ts_tablet_manager.cc:1428] T d61971a67ddd4c4d93d28ec056b2a852 P 3430624656cf4ddeb6381b272ae146e0: Time spent starting tablet: real 0.020s user 0.014s sys 0.006s
I20260620 23:13:08.574920 22866 tablet_bootstrap.cc:492] T d61971a67ddd4c4d93d28ec056b2a852 P 6305bba89eae4eeb9a2e37abb05c8cf6: No bootstrap required, opened a new log
I20260620 23:13:08.575280 22866 ts_tablet_manager.cc:1397] T d61971a67ddd4c4d93d28ec056b2a852 P 6305bba89eae4eeb9a2e37abb05c8cf6: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.006s
I20260620 23:13:08.582970 22866 raft_consensus.cc:348] T d61971a67ddd4c4d93d28ec056b2a852 P 6305bba89eae4eeb9a2e37abb05c8cf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6305bba89eae4eeb9a2e37abb05c8cf6" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 35263 } } peers { permanent_uuid: "3430624656cf4ddeb6381b272ae146e0" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40663 } } peers { permanent_uuid: "c390418022c145ac906c82aba8df0c17" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 36717 } }
I20260620 23:13:08.583290 22866 raft_consensus.cc:374] T d61971a67ddd4c4d93d28ec056b2a852 P 6305bba89eae4eeb9a2e37abb05c8cf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:08.583374 22866 raft_consensus.cc:729] T d61971a67ddd4c4d93d28ec056b2a852 P 6305bba89eae4eeb9a2e37abb05c8cf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6305bba89eae4eeb9a2e37abb05c8cf6, State: Initialized, Role: FOLLOWER
I20260620 23:13:08.583885 22866 consensus_queue.cc:260] T d61971a67ddd4c4d93d28ec056b2a852 P 6305bba89eae4eeb9a2e37abb05c8cf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6305bba89eae4eeb9a2e37abb05c8cf6" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 35263 } } peers { permanent_uuid: "3430624656cf4ddeb6381b272ae146e0" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40663 } } peers { permanent_uuid: "c390418022c145ac906c82aba8df0c17" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 36717 } }
I20260620 23:13:08.584896 22843 heartbeater.cc:503] Master 127.10.191.254:39701 was elected leader, sending a full tablet report...
I20260620 23:13:08.585552 22866 ts_tablet_manager.cc:1428] T d61971a67ddd4c4d93d28ec056b2a852 P 6305bba89eae4eeb9a2e37abb05c8cf6: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
W20260620 23:13:08.621713 22710 tablet.cc:2369] T d61971a67ddd4c4d93d28ec056b2a852 P c390418022c145ac906c82aba8df0c17: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:13:08.670804 22844 tablet.cc:2369] T d61971a67ddd4c4d93d28ec056b2a852 P 6305bba89eae4eeb9a2e37abb05c8cf6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:13:08.711330 22567 tablet.cc:2369] T d61971a67ddd4c4d93d28ec056b2a852 P 3430624656cf4ddeb6381b272ae146e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:13:08.840746 22870 raft_consensus.cc:482] T d61971a67ddd4c4d93d28ec056b2a852 P 6305bba89eae4eeb9a2e37abb05c8cf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:13:08.841033 22870 raft_consensus.cc:504] T d61971a67ddd4c4d93d28ec056b2a852 P 6305bba89eae4eeb9a2e37abb05c8cf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6305bba89eae4eeb9a2e37abb05c8cf6" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 35263 } } peers { permanent_uuid: "3430624656cf4ddeb6381b272ae146e0" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40663 } } peers { permanent_uuid: "c390418022c145ac906c82aba8df0c17" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 36717 } }
I20260620 23:13:08.842756 22870 leader_election.cc:288] T d61971a67ddd4c4d93d28ec056b2a852 P 6305bba89eae4eeb9a2e37abb05c8cf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3430624656cf4ddeb6381b272ae146e0 (127.10.191.193:40663), c390418022c145ac906c82aba8df0c17 (127.10.191.194:36717)
I20260620 23:13:08.849689 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:08.842977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:40663 (local address 127.10.191.195:33521)
0620 23:13:08.843259 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:08.843284 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:08.843464 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:08.843891 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:08.843916 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:08.843947 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:13:08.844153 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:08.844169 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.845489 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.845497 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:08.847501 (+ 2004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:08.847515 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.848954 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.848959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:08.849082 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:08.849106 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 23:13:08.849239 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 23:13:08.849370 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":23}
I20260620 23:13:08.849737 22860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:08.843214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:33521 (local address 127.10.191.193:40663)
0620 23:13:08.843446 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:08.843453 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:08.843476 (+ 23us) server_negotiation.cc:407] Connection header received
0620 23:13:08.843624 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:08.843629 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:08.843717 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:08.843837 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:13:08.844301 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.845318 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.847658 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.848765 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.848923 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:08.849016 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:08.849342 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 23:13:08.849485 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 23:13:08.850276 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:08.843165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:57743 (local address 127.10.191.194:36717)
0620 23:13:08.843409 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:08.843416 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:08.844207 (+ 791us) server_negotiation.cc:407] Connection header received
0620 23:13:08.844374 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:08.844380 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:08.844482 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:08.844578 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:13:08.845052 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.845898 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.848121 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.849340 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.849490 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:08.849551 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:08.849905 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 23:13:08.850058 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 23:13:08.850998 22871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:08.843521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.194:36717 (local address 127.10.191.195:57743)
0620 23:13:08.844104 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:08.844121 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:08.844250 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:08.844625 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:08.844629 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:08.844648 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:13:08.844908 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:08.844916 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.846043 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.846048 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:08.847971 (+ 1923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:08.847980 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.849531 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.849537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:08.849707 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:08.849749 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 23:13:08.850469 (+ 720us) client_negotiation.cc:240] Negotiation successful
0620 23:13:08.850598 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":1,"thread_start_us":282,"threads_started":1}
I20260620 23:13:08.850734 22521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d61971a67ddd4c4d93d28ec056b2a852" candidate_uuid: "6305bba89eae4eeb9a2e37abb05c8cf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3430624656cf4ddeb6381b272ae146e0" is_pre_election: true
I20260620 23:13:08.851346 22521 raft_consensus.cc:2454] T d61971a67ddd4c4d93d28ec056b2a852 P 3430624656cf4ddeb6381b272ae146e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6305bba89eae4eeb9a2e37abb05c8cf6 in term 0.
I20260620 23:13:08.852047 22734 leader_election.cc:302] T d61971a67ddd4c4d93d28ec056b2a852 P 6305bba89eae4eeb9a2e37abb05c8cf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3430624656cf4ddeb6381b272ae146e0, 6305bba89eae4eeb9a2e37abb05c8cf6; no voters:
I20260620 23:13:08.852074 22664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d61971a67ddd4c4d93d28ec056b2a852" candidate_uuid: "6305bba89eae4eeb9a2e37abb05c8cf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c390418022c145ac906c82aba8df0c17" is_pre_election: true
I20260620 23:13:08.852484 22870 raft_consensus.cc:2781] T d61971a67ddd4c4d93d28ec056b2a852 P 6305bba89eae4eeb9a2e37abb05c8cf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:13:08.852628 22870 raft_consensus.cc:482] T d61971a67ddd4c4d93d28ec056b2a852 P 6305bba89eae4eeb9a2e37abb05c8cf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:13:08.852602 22664 raft_consensus.cc:2454] T d61971a67ddd4c4d93d28ec056b2a852 P c390418022c145ac906c82aba8df0c17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6305bba89eae4eeb9a2e37abb05c8cf6 in term 0.
I20260620 23:13:08.852734 22870 raft_consensus.cc:3037] T d61971a67ddd4c4d93d28ec056b2a852 P 6305bba89eae4eeb9a2e37abb05c8cf6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:08.854946 22870 raft_consensus.cc:504] T d61971a67ddd4c4d93d28ec056b2a852 P 6305bba89eae4eeb9a2e37abb05c8cf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6305bba89eae4eeb9a2e37abb05c8cf6" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 35263 } } peers { permanent_uuid: "3430624656cf4ddeb6381b272ae146e0" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40663 } } peers { permanent_uuid: "c390418022c145ac906c82aba8df0c17" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 36717 } }
I20260620 23:13:08.855481 22870 leader_election.cc:288] T d61971a67ddd4c4d93d28ec056b2a852 P 6305bba89eae4eeb9a2e37abb05c8cf6 [CANDIDATE]: Term 1 election: Requested vote from peers 3430624656cf4ddeb6381b272ae146e0 (127.10.191.193:40663), c390418022c145ac906c82aba8df0c17 (127.10.191.194:36717)
I20260620 23:13:08.855990 22664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d61971a67ddd4c4d93d28ec056b2a852" candidate_uuid: "6305bba89eae4eeb9a2e37abb05c8cf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c390418022c145ac906c82aba8df0c17"
I20260620 23:13:08.856014 22521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d61971a67ddd4c4d93d28ec056b2a852" candidate_uuid: "6305bba89eae4eeb9a2e37abb05c8cf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3430624656cf4ddeb6381b272ae146e0"
I20260620 23:13:08.856200 22521 raft_consensus.cc:3037] T d61971a67ddd4c4d93d28ec056b2a852 P 3430624656cf4ddeb6381b272ae146e0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:08.856184 22664 raft_consensus.cc:3037] T d61971a67ddd4c4d93d28ec056b2a852 P c390418022c145ac906c82aba8df0c17 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:08.858263 22521 raft_consensus.cc:2454] T d61971a67ddd4c4d93d28ec056b2a852 P 3430624656cf4ddeb6381b272ae146e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6305bba89eae4eeb9a2e37abb05c8cf6 in term 1.
I20260620 23:13:08.858583 22664 raft_consensus.cc:2454] T d61971a67ddd4c4d93d28ec056b2a852 P c390418022c145ac906c82aba8df0c17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6305bba89eae4eeb9a2e37abb05c8cf6 in term 1.
I20260620 23:13:08.858682 22734 leader_election.cc:302] T d61971a67ddd4c4d93d28ec056b2a852 P 6305bba89eae4eeb9a2e37abb05c8cf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3430624656cf4ddeb6381b272ae146e0, 6305bba89eae4eeb9a2e37abb05c8cf6; no voters:
I20260620 23:13:08.859000 22870 raft_consensus.cc:2781] T d61971a67ddd4c4d93d28ec056b2a852 P 6305bba89eae4eeb9a2e37abb05c8cf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:13:08.859468 22870 raft_consensus.cc:686] T d61971a67ddd4c4d93d28ec056b2a852 P 6305bba89eae4eeb9a2e37abb05c8cf6 [term 1 LEADER]: Becoming Leader. State: Replica: 6305bba89eae4eeb9a2e37abb05c8cf6, State: Running, Role: LEADER
I20260620 23:13:08.859954 22870 consensus_queue.cc:237] T d61971a67ddd4c4d93d28ec056b2a852 P 6305bba89eae4eeb9a2e37abb05c8cf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6305bba89eae4eeb9a2e37abb05c8cf6" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 35263 } } peers { permanent_uuid: "3430624656cf4ddeb6381b272ae146e0" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40663 } } peers { permanent_uuid: "c390418022c145ac906c82aba8df0c17" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 36717 } }
I20260620 23:13:08.864821 22352 catalog_manager.cc:5495] T d61971a67ddd4c4d93d28ec056b2a852 P 6305bba89eae4eeb9a2e37abb05c8cf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6305bba89eae4eeb9a2e37abb05c8cf6 (127.10.191.195). New cstate: current_term: 1 leader_uuid: "6305bba89eae4eeb9a2e37abb05c8cf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6305bba89eae4eeb9a2e37abb05c8cf6" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 35263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3430624656cf4ddeb6381b272ae146e0" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c390418022c145ac906c82aba8df0c17" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 36717 } health_report { overall_health: UNKNOWN } } }
I20260620 23:13:08.892970 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:08.887376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50282 (local address 127.10.191.195:35263)
0620 23:13:08.887601 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:08.887607 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:08.887702 (+ 95us) server_negotiation.cc:407] Connection header received
0620 23:13:08.887946 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:08.887950 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:08.888024 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:08.888174 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:13:08.888610 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.889681 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.891091 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.891475 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.891570 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:08.892018 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:13:08.892474 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:13:08.892586 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:08.892683 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 23:13:08.892754 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 23:13:08.900950 22664 raft_consensus.cc:1261] T d61971a67ddd4c4d93d28ec056b2a852 P c390418022c145ac906c82aba8df0c17 [term 1 FOLLOWER]: Refusing update from remote peer 6305bba89eae4eeb9a2e37abb05c8cf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:13:08.901093 22521 raft_consensus.cc:1261] T d61971a67ddd4c4d93d28ec056b2a852 P 3430624656cf4ddeb6381b272ae146e0 [term 1 FOLLOWER]: Refusing update from remote peer 6305bba89eae4eeb9a2e37abb05c8cf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:13:08.901984 22872 consensus_queue.cc:1035] T d61971a67ddd4c4d93d28ec056b2a852 P 6305bba89eae4eeb9a2e37abb05c8cf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c390418022c145ac906c82aba8df0c17" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:13:08.902295 22870 consensus_queue.cc:1035] T d61971a67ddd4c4d93d28ec056b2a852 P 6305bba89eae4eeb9a2e37abb05c8cf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3430624656cf4ddeb6381b272ae146e0" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 40663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:13:08.916955 22877 mvcc.cc:205] Tried to move back new op lower bound from 7299060485728452608 to 7299060485574311936. Current Snapshot: MvccSnapshot[applied={T|T < 7299060485728452608}]
I20260620 23:13:08.926285 22352 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:13:08.926544 22352 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:13:08.930166 22352 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1d50e5942de04f3588a22df345770d4d: Sending DeleteTablet for 3 replicas of tablet d61971a67ddd4c4d93d28ec056b2a852
I20260620 23:13:08.931187 22778 tablet_service.cc:1514] Processing DeleteTablet for tablet d61971a67ddd4c4d93d28ec056b2a852 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:13:08 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:50276
I20260620 23:13:08.931248 22501 tablet_service.cc:1514] Processing DeleteTablet for tablet d61971a67ddd4c4d93d28ec056b2a852 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:13:08 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:48436
I20260620 23:13:08.931229 22644 tablet_service.cc:1514] Processing DeleteTablet for tablet d61971a67ddd4c4d93d28ec056b2a852 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:13:08 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:40606
I20260620 23:13:08.935130 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 22433
I20260620 23:13:08.947127 22884 tablet_replica.cc:330] T d61971a67ddd4c4d93d28ec056b2a852 P 6305bba89eae4eeb9a2e37abb05c8cf6: stopping tablet replica
I20260620 23:13:08.947649 22884 raft_consensus.cc:2229] T d61971a67ddd4c4d93d28ec056b2a852 P 6305bba89eae4eeb9a2e37abb05c8cf6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:13:08.948525 22884 raft_consensus.cc:2258] T d61971a67ddd4c4d93d28ec056b2a852 P 6305bba89eae4eeb9a2e37abb05c8cf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:13:08.950690 22884 ts_tablet_manager.cc:1905] T d61971a67ddd4c4d93d28ec056b2a852 P 6305bba89eae4eeb9a2e37abb05c8cf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:13:08.957732 22884 ts_tablet_manager.cc:1918] T d61971a67ddd4c4d93d28ec056b2a852 P 6305bba89eae4eeb9a2e37abb05c8cf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:13:08.957872 22884 log.cc:1196] T d61971a67ddd4c4d93d28ec056b2a852 P 6305bba89eae4eeb9a2e37abb05c8cf6: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-2/wal/wals/d61971a67ddd4c4d93d28ec056b2a852
I20260620 23:13:08.958346 22884 ts_tablet_manager.cc:1939] T d61971a67ddd4c4d93d28ec056b2a852 P 6305bba89eae4eeb9a2e37abb05c8cf6: Deleting consensus metadata
I20260620 23:13:08.960224 22339 catalog_manager.cc:4841] TS 6305bba89eae4eeb9a2e37abb05c8cf6 (127.10.191.195:35263): tablet d61971a67ddd4c4d93d28ec056b2a852 (table test-table [id=4d466850b1714bf5a2ff189b8e06ab36]) successfully deleted
I20260620 23:13:08.968104 22885 tablet_replica.cc:330] T d61971a67ddd4c4d93d28ec056b2a852 P 3430624656cf4ddeb6381b272ae146e0: stopping tablet replica
I20260620 23:13:08.968725 22885 raft_consensus.cc:2229] T d61971a67ddd4c4d93d28ec056b2a852 P 3430624656cf4ddeb6381b272ae146e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:13:08.969266 22885 raft_consensus.cc:2258] T d61971a67ddd4c4d93d28ec056b2a852 P 3430624656cf4ddeb6381b272ae146e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:13:08.969830 22886 tablet_replica.cc:330] T d61971a67ddd4c4d93d28ec056b2a852 P c390418022c145ac906c82aba8df0c17: stopping tablet replica
I20260620 23:13:08.971520 22885 ts_tablet_manager.cc:1905] T d61971a67ddd4c4d93d28ec056b2a852 P 3430624656cf4ddeb6381b272ae146e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:13:08.972028 22886 raft_consensus.cc:2229] T d61971a67ddd4c4d93d28ec056b2a852 P c390418022c145ac906c82aba8df0c17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:13:08.972572 22886 raft_consensus.cc:2258] T d61971a67ddd4c4d93d28ec056b2a852 P c390418022c145ac906c82aba8df0c17 [term 1 FOLLOWER]: Raft consensus is shut down!
Jun 20 23:13:08 dist-test-slave-jkbq krb5kdc[21877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.193@KRBTEST.COM
I20260620 23:13:08.979172 22885 ts_tablet_manager.cc:1918] T d61971a67ddd4c4d93d28ec056b2a852 P 3430624656cf4ddeb6381b272ae146e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:13:08.979342 22885 log.cc:1196] T d61971a67ddd4c4d93d28ec056b2a852 P 3430624656cf4ddeb6381b272ae146e0: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-0/wal/wals/d61971a67ddd4c4d93d28ec056b2a852
I20260620 23:13:08.979892 22885 ts_tablet_manager.cc:1939] T d61971a67ddd4c4d93d28ec056b2a852 P 3430624656cf4ddeb6381b272ae146e0: Deleting consensus metadata
I20260620 23:13:08.981839 22340 catalog_manager.cc:4841] TS 3430624656cf4ddeb6381b272ae146e0 (127.10.191.193:40663): tablet d61971a67ddd4c4d93d28ec056b2a852 (table test-table [id=4d466850b1714bf5a2ff189b8e06ab36]) successfully deleted
I20260620 23:13:08.983059 22886 ts_tablet_manager.cc:1905] T d61971a67ddd4c4d93d28ec056b2a852 P c390418022c145ac906c82aba8df0c17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:13:08.986444 22860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:08.936805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48452 (local address 127.10.191.193:40663)
0620 23:13:08.937032 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:08.937038 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:08.967397 (+ 30359us) server_negotiation.cc:407] Connection header received
0620 23:13:08.967624 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:08.967632 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:08.967734 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:08.967858 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:08.969796 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.970924 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.972824 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:08.973510 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:08.973618 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:08.979274 (+ 5656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:08.979289 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:08.979306 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:08.979353 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:08.983234 (+ 3881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:08.983906 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:08.983915 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:08.983918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:08.984112 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:08.984596 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:08.984602 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:08.984605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:08.984945 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:08.985259 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:08.985801 (+ 542us) server_negotiation.cc:299] Negotiation successful
0620 23:13:08.986011 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:13:08.987145 22561 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:13:08.990226 22886 ts_tablet_manager.cc:1918] T d61971a67ddd4c4d93d28ec056b2a852 P c390418022c145ac906c82aba8df0c17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:13:08.990404 22886 log.cc:1196] T d61971a67ddd4c4d93d28ec056b2a852 P c390418022c145ac906c82aba8df0c17: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781997042893763-11007-0/minicluster-data/ts-1/wal/wals/d61971a67ddd4c4d93d28ec056b2a852
I20260620 23:13:08.991007 22886 ts_tablet_manager.cc:1939] T d61971a67ddd4c4d93d28ec056b2a852 P c390418022c145ac906c82aba8df0c17: Deleting consensus metadata
I20260620 23:13:08.992755 22338 catalog_manager.cc:4841] TS c390418022c145ac906c82aba8df0c17 (127.10.191.194:36717): tablet d61971a67ddd4c4d93d28ec056b2a852 (table test-table [id=4d466850b1714bf5a2ff189b8e06ab36]) successfully deleted
I20260620 23:13:09.115352 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 22433
I20260620 23:13:09.132424 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 22572
Jun 20 23:13:09 dist-test-slave-jkbq krb5kdc[21877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.194@KRBTEST.COM
I20260620 23:13:09.151170 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:09.133241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40620 (local address 127.10.191.194:36717)
0620 23:13:09.133416 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:09.133422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:09.133442 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:13:09.133674 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:09.133681 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:09.133762 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:09.133859 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:09.135582 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:09.136415 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:09.138329 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:09.138786 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:09.139113 (+ 327us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:09.144339 (+ 5226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:09.144355 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:09.144371 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:09.144413 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:09.147791 (+ 3378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:09.148696 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:09.148705 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:09.148707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:09.148864 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:09.149308 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:09.149314 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:09.149316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:09.149771 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:09.150095 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:09.150469 (+ 374us) server_negotiation.cc:299] Negotiation successful
0620 23:13:09.150752 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:13:09.151883 22704 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:13:09.239574 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 22572
I20260620 23:13:09.261426 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 22715
Jun 20 23:13:09 dist-test-slave-jkbq krb5kdc[21877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.195@KRBTEST.COM
I20260620 23:13:09.279670 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:09.262310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50296 (local address 127.10.191.195:35263)
0620 23:13:09.262524 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:09.262528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:09.262548 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:13:09.262747 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:09.262751 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:09.262809 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:09.262956 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:09.264725 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:09.265556 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:09.267681 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:09.268115 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:09.268216 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:09.273378 (+ 5162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:09.273390 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:09.273409 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:09.273493 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:09.276587 (+ 3094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:09.277478 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:09.277486 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:09.277489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:09.277636 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:09.278041 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:09.278044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:09.278046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:09.278369 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:09.278705 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:09.279065 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 23:13:09.279320 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 23:13:09.280324 22838 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:13:09.402035 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 22715
I20260620 23:13:09.428496 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 22321
I20260620 23:13:09.429765 22383 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:13:09.539866 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 22321
I20260620 23:13:09.560670 11007 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 23:13:09.896229 11007 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 23:13:09.896517 11007 mini_ranger.cc:66] Stopping Ranger...
I20260620 23:13:10.242352 11007 mini_ranger.cc:68] Stopped Ranger
2026-06-20 23:13:10.242 UTC [21907] LOG: received smart shutdown request
2026-06-20 23:13:10.246 UTC [21907] LOG: background worker "logical replication launcher" (PID 21916) exited with exit code 1
2026-06-20 23:13:10.247 UTC [21911] LOG: shutting down
2026-06-20 23:13:10.406 UTC [21907] LOG: database system is shut down
2026-06-20T23:13:10Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49517 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:13:10 dist-test-slave-jkbq krb5kdc[22900](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:13:10 dist-test-slave-jkbq krb5kdc[22900](info): set up 2 sockets
Jun 20 23:13:10 dist-test-slave-jkbq krb5kdc[22900](info): commencing operation
krb5kdc: starting...
W20260620 23:13:11.643322 11007 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Jun 20 23:13:11 dist-test-slave-jkbq krb5kdc[22900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:13:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:13:11Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.10.191.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.10.191.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.10.191.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/rangeradmin_127.10.191.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.10.191.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/rangeradmin_127.10.191.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.10.191.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.10.191.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.10.191.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/rangerlookup_127.10.191.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.10.191.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/rangerlookup_127.10.191.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.191.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.191.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.191.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/HTTP_127.10.191.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.191.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/HTTP_127.10.191.212@KRBTEST.COM.keytab.
I20260620 23:13:11.945694 11007 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskpRJWBp/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/postgres -l logfile start
2026-06-20 23:13:14.336 UTC [22930] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 23:13:14.337 UTC [22930] LOG: listening on IPv4 address "127.10.191.212", port 55613
2026-06-20 23:13:14.340 UTC [22930] LOG: listening on Unix socket "/tmp/.s.PGSQL.55613"
2026-06-20 23:13:14.361 UTC [22933] LOG: database system was shut down at 2026-06-20 23:13:13 UTC
2026-06-20 23:13:14.365 UTC [22930] LOG: database system is ready to accept connections
I20260620 23:13:15.383030 11007 mini_postgres.cc:96] Postgres bound to 55613
2026-06-20 23:13:15.399 UTC [22941] FATAL: database "slave" does not exist
127.10.191.212:55613 - accepting connections
I20260620 23:13:15.400125 11007 mini_ranger.cc:161] Starting Ranger...
I20260620 23:13:15.424125 11007 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 23:13:15.891994 11007 mini_ranger.cc:87] Created ranger Postgres database
I20260620 23:13:15.892131 11007 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ranger-admin
2026-06-20 23:13:16,169 [I] DB FLAVOR :POSTGRES
2026-06-20 23:13:16,170 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 23:13:16,170 [I] Checking connection..
2026-06-20 23:13:16,170 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.191.212:55613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 23:13:16,415 [I] Checking connection passed.
2026-06-20 23:13:16,416 [I] --------- Verifying version history table ---------
2026-06-20 23:13:16,416 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.191.212:55613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 23:13:16,676 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 23:13:16,676 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.191.212:55613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 23:13:16,924 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 23:13:16,924 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 23:13:16,925 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.191.212:55613/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 23:13:17.170 UTC [23005] WARNING: there is no transaction in progress
2026-06-20 23:13:17,176 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 23:13:17,177 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.191.212:55613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 23:13:17,433 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 23:13:17,433 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 23:13:17,434 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.191.212:55613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 23:13:17,655 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.191.212:55613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 23:13:17,892 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.191.212:55613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-jkbq.c.gcp-upstream.internal','N') ;"
2026-06-20 23:13:18,113 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.191.212:55613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 23:13:18,374 [I] Table x_portal_user does not exist in database ranger
2026-06-20 23:13:18,374 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.191.212:55613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 23:13:18,623 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 23:13:18,624 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.191.212:55613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 23:13:18,880 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.191.212:55613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 23:13:19,125 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 23:13:19,125 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.191.212:55613/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 23:13:20.063 UTC [23113] WARNING: there is no transaction in progress
2026-06-20 23:13:20.078 UTC [23113] WARNING: there is no transaction in progress
2026-06-20 23:13:20.093 UTC [23113] WARNING: there is no transaction in progress
2026-06-20 23:13:20.107 UTC [23113] WARNING: there is no transaction in progress
2026-06-20 23:13:20.123 UTC [23113] WARNING: there is no transaction in progress
2026-06-20 23:13:20.141 UTC [23113] WARNING: there is no transaction in progress
2026-06-20 23:13:20.200 UTC [23113] WARNING: there is no transaction in progress
2026-06-20 23:13:20.206 UTC [23113] WARNING: there is no transaction in progress
2026-06-20 23:13:20.219 UTC [23113] WARNING: there is no transaction in progress
2026-06-20 23:13:20.230 UTC [23113] WARNING: there is no transaction in progress
2026-06-20 23:13:20.240 UTC [23113] WARNING: there is no transaction in progress
2026-06-20 23:13:20.252 UTC [23113] WARNING: there is no transaction in progress
2026-06-20 23:13:20.261 UTC [23113] WARNING: there is no transaction in progress
2026-06-20 23:13:20.267 UTC [23113] WARNING: there is no transaction in progress
2026-06-20 23:13:20.273 UTC [23113] WARNING: there is no transaction in progress
2026-06-20 23:13:20.279 UTC [23113] WARNING: there is no transaction in progress
2026-06-20 23:13:20,869 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 23:13:20,869 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.191.212:55613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 23:13:21,112 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 23:13:21,112 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.191.212:55613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 23:13:21,393 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 23:13:21,394 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.191.212:55613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 23:13:21,636 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 23:13:21,637 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.191.212:55613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 23:13:21,869 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.191.212:55613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 23:13:22,126 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.191.212:55613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-jkbq.c.gcp-upstream.internal';"
2026-06-20 23:13:22,351 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 23:13:22,351 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.191.212:55613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 23:13:22,573 [I] DB_PATCHES have already been applied
I20260620 23:13:22.579664 11007 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ranger-admin:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskpRJWBp/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 23:13:22.579743 11007 mini_ranger.cc:193] Using host: 127.10.191.212
I20260620 23:13:22.593578 11007 mini_ranger.cc:227] Ranger admin URL: http://127.10.191.212:46365
Jun 20, 2026 11:13:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 11:13:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 11:13:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 11:13:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 11:13:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 11:13:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 11:13:23 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 23:13:23.733605 11007 mini_ranger.cc:160] Time spent starting Ranger: real 8.333s user 0.002s sys 0.032s
Jun 20, 2026 11:13:43 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 11:13:43 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 11:13:43 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 11:13:43 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 11:13:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 11:13:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 11:13:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 11:13:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 11:13:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 11:13:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 11:13:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 11:13:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 11:13:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 11:13:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 11:13:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 11:13:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 11:13:43 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 11:13:44 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 23:13:44.637645 11007 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.10.191.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.10.191.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.10.191.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/rangerkms_127.10.191.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.10.191.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/rangerkms_127.10.191.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.191.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.10.191.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.191.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/HTTP_127.10.191.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.191.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/HTTP_127.10.191.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 23:13:44 dist-test-slave-jkbq krb5kdc[22900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997224, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 23:13:44.824190 11007 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 23:13:44.843961 11007 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 23:13:45.691358 11007 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 23:13:45.691455 11007 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ranger-kms
I20260620 23:13:45.691486 11007 mini_ranger_kms.cc:227] Using postgres at 127.10.191.212:55613
2026-06-20 23:13:46,045 [I] DB FLAVOR :POSTGRES
2026-06-20 23:13:46,045 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 23:13:46,045 [I] Checking connection
2026-06-20 23:13:46,046 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.191.212:55613/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 23:13:46,273 [I] connection success
2026-06-20 23:13:46,274 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 23:13:46,274 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 23:13:46,274 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.191.212:55613/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 23:13:46,526 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 23:13:46,526 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 23:13:46,526 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 23:13:46,526 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.191.212:55613/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 23:13:46,803 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 23:13:47.258368 11007 mini_ranger_kms.cc:310] Created kms service
I20260620 23:13:47.453157 11007 mini_ranger_kms.cc:326] Created kudu user
I20260620 23:13:47.498493 11007 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 23:13:47 dist-test-slave-jkbq krb5kdc[22900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997224, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.10.191.212@KRBTEST.COM
I20260620 23:13:47.892239 11007 mini_ranger_kms.cc:384] Added ranger policy
I20260620 23:13:47.892598 11007 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ranger-kms:/tmp/dist-test-taskpRJWBp/build/asan/bin/postgresql.jar:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskpRJWBp/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 23:13:47.892652 11007 mini_ranger_kms.cc:240] Using host: 127.10.191.212
I20260620 23:13:47.912572 11007 mini_ranger_kms.cc:276] Ranger KMS PID: 23293
I20260620 23:13:47.912724 11007 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.10.191.212:48067
Jun 20, 2026 11:13:49 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 11:13:49 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 11:13:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 11:13:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 11:13:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 11:13:49 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 23:13:50.238827 11007 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.415s user 0.056s sys 0.311s
I20260620 23:13:50.239112 11007 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 23:13:50.239285 11007 mini_ranger_kms.cc:401] 127.10.191.212:48067/kms/v1/keys
Jun 20 23:13:50 dist-test-slave-jkbq krb5kdc[22900](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781997230, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.191.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 23:13:50 dist-test-slave-jkbq krb5kdc[22900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997230, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.191.212@KRBTEST.COM for HTTP/127.10.191.212@KRBTEST.COM
Jun 20 23:13:51 dist-test-slave-jkbq krb5kdc[22900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997230, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.191.212@KRBTEST.COM for HTTP/127.10.191.212@KRBTEST.COM
Jun 20 23:13:53 dist-test-slave-jkbq krb5kdc[22900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.10.191.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
I20260620 23:13:53.609499 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:43645
--webserver_interface=127.10.191.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:33001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:43645
--ranger_config_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.191.212:48067/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:13:53.948553 23344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:53.948853 23344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:53.949047 23344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:53.956632 23344 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:13:53.956750 23344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:53.956842 23344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:53.956952 23344 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:13:53.957026 23344 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:13:53.967675 23344 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:33001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/master-0/wal
--ranger_kms_url=127.10.191.212:48067/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:43645
--ranger_config_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:43645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:13:53.969081 23344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:53.970564 23344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:53.982887 23349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:53.982909 23352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:53.983003 23350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:53.983948 23344 server_base.cc:1034] running on GCE node
I20260620 23:13:53.984751 23344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:53.986258 23344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:53.987519 23344 hybrid_clock.cc:648] HybridClock initialized: now 1781997233987461 us; error 59 us; skew 500 ppm
I20260620 23:13:53.987893 23344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:13:53 dist-test-slave-jkbq krb5kdc[22900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997233, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:53.991122 23344 init.cc:377] Logged in from keytab as kudu/127.10.191.254@KRBTEST.COM (short username kudu)
I20260620 23:13:53.992569 23344 webserver.cc:460] Webserver started at http://127.10.191.254:41543/ using document root <none> and password file <none>
I20260620 23:13:53.993206 23344 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:53.993338 23344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:53.993645 23344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:13:53 dist-test-slave-jkbq krb5kdc[22900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997233, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for HTTP/127.10.191.212@KRBTEST.COM
I20260620 23:13:54.056882 23344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "9056d50dd8d542c7b9812cc051b61231"
format_stamp: "Formatted at 2026-06-20 23:13:54 on dist-test-slave-jkbq"
server_key: "a684d13650b957bc111aa31ea30d44f8"
server_key_iv: "a7c461f7bc4147fafd768e5201b24c62"
server_key_version: "kuduclusterkey@0"
I20260620 23:13:54.057781 23344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "9056d50dd8d542c7b9812cc051b61231"
format_stamp: "Formatted at 2026-06-20 23:13:54 on dist-test-slave-jkbq"
server_key: "a684d13650b957bc111aa31ea30d44f8"
server_key_iv: "a7c461f7bc4147fafd768e5201b24c62"
server_key_version: "kuduclusterkey@0"
I20260620 23:13:54.063462 23344 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260620 23:13:54.084576 23359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:54.086229 23344 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 23:13:54.086449 23344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "9056d50dd8d542c7b9812cc051b61231"
format_stamp: "Formatted at 2026-06-20 23:13:54 on dist-test-slave-jkbq"
server_key: "a684d13650b957bc111aa31ea30d44f8"
server_key_iv: "a7c461f7bc4147fafd768e5201b24c62"
server_key_version: "kuduclusterkey@0"
I20260620 23:13:54.086726 23344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:54.105677 23344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:54.110577 23344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:54.110958 23344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:54.132114 23344 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:43645
I20260620 23:13:54.132160 23410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:43645 every 8 connection(s)
I20260620 23:13:54.133585 23344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:13:54.137485 23411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:13:54.143185 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 23344
I20260620 23:13:54.143417 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:13:54.144114 11007 external_mini_cluster.cc:1428] Setting key 8caefb1c7a937d963b30893489276ed2
I20260620 23:13:54.149333 23411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9056d50dd8d542c7b9812cc051b61231: Bootstrap starting.
I20260620 23:13:54.153334 23411 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9056d50dd8d542c7b9812cc051b61231: Neither blocks nor log segments found. Creating new log.
I20260620 23:13:54.154532 23411 log.cc:824] T 00000000000000000000000000000000 P 9056d50dd8d542c7b9812cc051b61231: Log is configured to *not* fsync() on all Append() calls
I20260620 23:13:54.157768 23411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9056d50dd8d542c7b9812cc051b61231: No bootstrap required, opened a new log
Jun 20 23:13:54 dist-test-slave-jkbq krb5kdc[22900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:13:54.167703 23411 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9056d50dd8d542c7b9812cc051b61231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9056d50dd8d542c7b9812cc051b61231" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43645 } }
I20260620 23:13:54.168160 23411 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9056d50dd8d542c7b9812cc051b61231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:13:54.168264 23411 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9056d50dd8d542c7b9812cc051b61231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9056d50dd8d542c7b9812cc051b61231, State: Initialized, Role: FOLLOWER
I20260620 23:13:54.168953 23411 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9056d50dd8d542c7b9812cc051b61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9056d50dd8d542c7b9812cc051b61231" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43645 } }
I20260620 23:13:54.169220 23411 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9056d50dd8d542c7b9812cc051b61231 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:13:54.169394 23414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:54.146310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55768 (local address 127.10.191.254:43645)
0620 23:13:54.147188 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:54.147204 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:54.147255 (+ 51us) server_negotiation.cc:407] Connection header received
0620 23:13:54.149626 (+ 2371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:54.149654 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:54.150171 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:54.150605 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:54.151916 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:54.153901 (+ 1985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:54.155616 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:54.156589 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:54.156767 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:54.160621 (+ 3854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:54.160632 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:54.160653 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:54.160701 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:54.164710 (+ 4009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:54.165396 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:54.165404 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:54.165413 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:54.165590 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:54.166007 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:54.166012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:54.166015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:54.166749 (+ 734us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:54.167106 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:54.167602 (+ 496us) server_negotiation.cc:299] Negotiation successful
0620 23:13:54.168035 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":351,"threads_started":1}
I20260620 23:13:54.169390 23411 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9056d50dd8d542c7b9812cc051b61231 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:13:54.169610 23411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9056d50dd8d542c7b9812cc051b61231 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:13:54.172052 23411 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9056d50dd8d542c7b9812cc051b61231 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9056d50dd8d542c7b9812cc051b61231" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43645 } }
I20260620 23:13:54.172557 23411 leader_election.cc:302] T 00000000000000000000000000000000 P 9056d50dd8d542c7b9812cc051b61231 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9056d50dd8d542c7b9812cc051b61231; no voters:
I20260620 23:13:54.173103 23411 leader_election.cc:288] T 00000000000000000000000000000000 P 9056d50dd8d542c7b9812cc051b61231 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:13:54.173243 23416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9056d50dd8d542c7b9812cc051b61231 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:13:54.174686 23416 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9056d50dd8d542c7b9812cc051b61231 [term 1 LEADER]: Becoming Leader. State: Replica: 9056d50dd8d542c7b9812cc051b61231, State: Running, Role: LEADER
I20260620 23:13:54.175297 23411 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9056d50dd8d542c7b9812cc051b61231 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:13:54.175225 23416 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9056d50dd8d542c7b9812cc051b61231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9056d50dd8d542c7b9812cc051b61231" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43645 } }
I20260620 23:13:54.180663 23418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9056d50dd8d542c7b9812cc051b61231 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9056d50dd8d542c7b9812cc051b61231. Latest consensus state: current_term: 1 leader_uuid: "9056d50dd8d542c7b9812cc051b61231" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9056d50dd8d542c7b9812cc051b61231" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43645 } } }
I20260620 23:13:54.180636 23417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9056d50dd8d542c7b9812cc051b61231 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9056d50dd8d542c7b9812cc051b61231" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9056d50dd8d542c7b9812cc051b61231" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43645 } } }
I20260620 23:13:54.181111 23418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9056d50dd8d542c7b9812cc051b61231 [sys.catalog]: This master's current role is: LEADER
I20260620 23:13:54.181191 23417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9056d50dd8d542c7b9812cc051b61231 [sys.catalog]: This master's current role is: LEADER
I20260620 23:13:54.187368 23426 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:13:54.192780 23426 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:13:54.193420 23411 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 23:13:54.211536 23426 catalog_manager.cc:1347] Generated new cluster ID: 5a830c583eda49afb1b052ee5a21259c
I20260620 23:13:54.211700 23426 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:13:54.235985 23426 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:13:54.237450 23426 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:13:54.246829 23426 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9056d50dd8d542c7b9812cc051b61231: Generated new TSK 0
I20260620 23:13:54.248162 23426 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.191.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab.
Entry for principal kudu/127.10.191.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab.
I20260620 23:13:55.020844 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.191.212:48067/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.193:0
--local_ip_for_outbound_sockets=127.10.191.193
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:43645
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:33001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:13:55 dist-test-slave-jkbq krb5kdc[22900](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781997235, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 23:13:55.442605 23456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:55.443032 23456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:55.443248 23456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:55.454800 23456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:55.454957 23456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:55.455195 23456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.193
I20260620 23:13:55.470752 23456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:33001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.10.191.212:48067/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:43645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.193
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:13:55.472530 23456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:55.474324 23456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:55.489818 23466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:55.490765 23467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:55.492692 23456 server_base.cc:1034] running on GCE node
W20260620 23:13:55.493342 23469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:55.493932 23456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:55.494930 23456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:55.496171 23456 hybrid_clock.cc:648] HybridClock initialized: now 1781997235496075 us; error 83 us; skew 500 ppm
I20260620 23:13:55.496587 23456 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:13:55 dist-test-slave-jkbq krb5kdc[22900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997235, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:55.504096 23456 init.cc:377] Logged in from keytab as kudu/127.10.191.193@KRBTEST.COM (short username kudu)
I20260620 23:13:55.506115 23456 webserver.cc:460] Webserver started at http://127.10.191.193:46629/ using document root <none> and password file <none>
I20260620 23:13:55.507113 23456 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:55.507253 23456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:55.507642 23456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:13:55 dist-test-slave-jkbq krb5kdc[22900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997235, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for HTTP/127.10.191.212@KRBTEST.COM
I20260620 23:13:55.524587 23456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-0/data/instance:
uuid: "fc11adde099c47558865f18fbde4be49"
format_stamp: "Formatted at 2026-06-20 23:13:55 on dist-test-slave-jkbq"
server_key: "e9ac9139e39c0e00d9465980f622c3c9"
server_key_iv: "cd61d57db488327cd3a7c0b9ba2762c0"
server_key_version: "kuduclusterkey@0"
I20260620 23:13:55.525705 23456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance:
uuid: "fc11adde099c47558865f18fbde4be49"
format_stamp: "Formatted at 2026-06-20 23:13:55 on dist-test-slave-jkbq"
server_key: "e9ac9139e39c0e00d9465980f622c3c9"
server_key_iv: "cd61d57db488327cd3a7c0b9ba2762c0"
server_key_version: "kuduclusterkey@0"
I20260620 23:13:55.536950 23456 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.002s sys 0.007s
I20260620 23:13:55.572170 23476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:55.574225 23456 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.000s sys 0.009s
I20260620 23:13:55.574429 23456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-0/wal
uuid: "fc11adde099c47558865f18fbde4be49"
format_stamp: "Formatted at 2026-06-20 23:13:55 on dist-test-slave-jkbq"
server_key: "e9ac9139e39c0e00d9465980f622c3c9"
server_key_iv: "cd61d57db488327cd3a7c0b9ba2762c0"
server_key_version: "kuduclusterkey@0"
I20260620 23:13:55.574702 23456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:55.622956 23456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:55.628312 23456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:55.628664 23456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:55.635953 23456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:13:55.643990 23456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:55.644088 23456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:55.644222 23456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:55.644295 23456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:13:55 dist-test-slave-jkbq krb5kdc[22900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997235, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for HTTP/127.10.191.212@KRBTEST.COM
I20260620 23:13:56.039244 23456 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.193:39305
I20260620 23:13:56.039283 23588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.193:39305 every 8 connection(s)
I20260620 23:13:56.040599 23456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
I20260620 23:13:56.048861 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 23456
I20260620 23:13:56.049074 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance
I20260620 23:13:56.049820 11007 external_mini_cluster.cc:1428] Setting key c386bb13c9b6242af36c73aadc08e9e3
Jun 20 23:13:56 dist-test-slave-jkbq krb5kdc[22900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997235, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.191.194@KRBTEST.COM; defaulting to no policy
I20260620 23:13:56.085485 23591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:56.045582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:44319 (local address 127.10.191.254:43645)
0620 23:13:56.054206 (+ 8624us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:56.054216 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:56.062919 (+ 8703us) server_negotiation.cc:407] Connection header received
0620 23:13:56.064780 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:56.064789 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:56.064892 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:56.064977 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:56.067472 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:56.068741 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:56.071394 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:56.072183 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:56.072300 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:56.077352 (+ 5052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:56.077362 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:56.077370 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:56.077419 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:56.081525 (+ 4106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:56.082668 (+ 1143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:56.082673 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:56.082678 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:56.082837 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:56.083396 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:56.083400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:56.083403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:56.083788 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:56.083907 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:56.084856 (+ 949us) server_negotiation.cc:299] Negotiation successful
0620 23:13:56.085072 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8436,"thread_start_us":8632,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.194@KRBTEST.COM" created.
I20260620 23:13:56.089114 23592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:56.046366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43645 (local address 127.10.191.193:44319)
0620 23:13:56.060397 (+ 14031us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:56.060460 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:56.064422 (+ 3962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:56.065613 (+ 1191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:56.065625 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:56.066341 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:56.067273 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:56.067299 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:56.068942 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:56.068951 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:56.071226 (+ 2275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:56.071245 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:56.072676 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:56.072685 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:56.073035 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:56.073878 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:56.073912 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:56.077062 (+ 3150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:56.081833 (+ 4771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:56.081842 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:56.081859 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:56.082517 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:56.083027 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:56.083033 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:56.083036 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:56.083272 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:56.084009 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:56.084017 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:56.084447 (+ 430us) client_negotiation.cc:769] Sending connection context
0620 23:13:56.086946 (+ 2499us) client_negotiation.cc:240] Negotiation successful
0620 23:13:56.087570 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13649,"thread_start_us":16448,"threads_started":1}
I20260620 23:13:56.093725 23589 heartbeater.cc:347] Connected to a master server at 127.10.191.254:43645
I20260620 23:13:56.094046 23589 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:56.094882 23589 heartbeater.cc:511] Master 127.10.191.254:43645 requested a full tablet report, sending...
I20260620 23:13:56.097386 23376 ts_manager.cc:194] Registered new tserver with Master: fc11adde099c47558865f18fbde4be49 (127.10.191.193:39305)
I20260620 23:13:56.099287 23376 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.193@KRBTEST.COM'} at 127.10.191.193:44319
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab.
Entry for principal kudu/127.10.191.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab.
I20260620 23:13:56.127789 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.191.212:48067/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.194:0
--local_ip_for_outbound_sockets=127.10.191.194
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:43645
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:33001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:13:56 dist-test-slave-jkbq krb5kdc[22900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997235, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for HTTP/127.10.191.212@KRBTEST.COM
W20260620 23:13:56.572328 23595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:56.572577 23595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:56.572754 23595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:56.580148 23595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:56.580263 23595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:56.580440 23595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.194
I20260620 23:13:56.590016 23595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:33001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.10.191.212:48067/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:43645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.194
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:13:56.591295 23595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:56.592692 23595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:56.605388 23609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:56.605466 23610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:56.605491 23612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:56.606397 23595 server_base.cc:1034] running on GCE node
I20260620 23:13:56.607062 23595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:56.608003 23595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:56.609220 23595 hybrid_clock.cc:648] HybridClock initialized: now 1781997236609176 us; error 34 us; skew 500 ppm
I20260620 23:13:56.609525 23595 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:13:56 dist-test-slave-jkbq krb5kdc[22900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997236, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:56.612519 23595 init.cc:377] Logged in from keytab as kudu/127.10.191.194@KRBTEST.COM (short username kudu)
I20260620 23:13:56.613818 23595 webserver.cc:460] Webserver started at http://127.10.191.194:42327/ using document root <none> and password file <none>
I20260620 23:13:56.614459 23595 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:56.614598 23595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:56.614969 23595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:13:56 dist-test-slave-jkbq krb5kdc[22900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997236, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for HTTP/127.10.191.212@KRBTEST.COM
I20260620 23:13:56.628350 23595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-1/data/instance:
uuid: "b20faef2484548919509bf90c9ab7eb5"
format_stamp: "Formatted at 2026-06-20 23:13:56 on dist-test-slave-jkbq"
server_key: "ed78cc40d9e1a7b898de81b60df57804"
server_key_iv: "bba1428df34255738324560538956dab"
server_key_version: "kuduclusterkey@0"
I20260620 23:13:56.629207 23595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance:
uuid: "b20faef2484548919509bf90c9ab7eb5"
format_stamp: "Formatted at 2026-06-20 23:13:56 on dist-test-slave-jkbq"
server_key: "ed78cc40d9e1a7b898de81b60df57804"
server_key_iv: "bba1428df34255738324560538956dab"
server_key_version: "kuduclusterkey@0"
I20260620 23:13:56.634836 23595 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 23:13:56.651726 23619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:56.653395 23595 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:13:56.653561 23595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-1/wal
uuid: "b20faef2484548919509bf90c9ab7eb5"
format_stamp: "Formatted at 2026-06-20 23:13:56 on dist-test-slave-jkbq"
server_key: "ed78cc40d9e1a7b898de81b60df57804"
server_key_iv: "bba1428df34255738324560538956dab"
server_key_version: "kuduclusterkey@0"
I20260620 23:13:56.653802 23595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:56.676223 23595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:56.680855 23595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:56.681182 23595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:56.682354 23595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:13:56.683844 23595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:56.683948 23595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:56.684104 23595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:56.684176 23595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:56.720537 23595 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.194:43103
I20260620 23:13:56.720625 23731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.194:43103 every 8 connection(s)
I20260620 23:13:56.721861 23595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
I20260620 23:13:56.729830 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 23595
I20260620 23:13:56.730028 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance
I20260620 23:13:56.730476 11007 external_mini_cluster.cc:1428] Setting key c752e66af3cb8d92b2f4ab9c27df522e
Jun 20 23:13:56 dist-test-slave-jkbq krb5kdc[22900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997236, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:13:56.743510 23734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:56.724707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:39957 (local address 127.10.191.254:43645)
0620 23:13:56.725201 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:56.725207 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:56.726164 (+ 957us) server_negotiation.cc:407] Connection header received
0620 23:13:56.728563 (+ 2399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:56.728567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:56.728644 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:56.728740 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:56.730650 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:56.731843 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:56.733526 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:56.733995 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:56.734124 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:56.737315 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:56.737320 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:56.737329 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:56.737359 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:56.739893 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:56.740642 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:56.740651 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:56.740655 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:56.740800 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:56.741328 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:56.741335 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:56.741340 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:56.741713 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:56.741842 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:56.742736 (+ 894us) server_negotiation.cc:299] Negotiation successful
0620 23:13:56.743016 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":262,"threads_started":1}
I20260620 23:13:56.744594 23735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:56.725187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43645 (local address 127.10.191.194:39957)
0620 23:13:56.725977 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:56.726040 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:56.728245 (+ 2205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:56.729027 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:56.729039 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:56.729643 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:56.730437 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:56.730461 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:56.732032 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:56.732037 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:56.733375 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:56.733396 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:56.734204 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:56.734210 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:56.734452 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:56.735144 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:56.735166 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:56.737113 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:56.740040 (+ 2927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:56.740048 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:56.740064 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:56.740470 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:56.740944 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:56.740949 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:56.740953 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:56.741206 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:56.741867 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:56.741875 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:56.742284 (+ 409us) client_negotiation.cc:769] Sending connection context
0620 23:13:56.742608 (+ 324us) client_negotiation.cc:240] Negotiation successful
0620 23:13:56.743188 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":304,"threads_started":1}
I20260620 23:13:56.746146 23732 heartbeater.cc:347] Connected to a master server at 127.10.191.254:43645
I20260620 23:13:56.746475 23732 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:56.747347 23732 heartbeater.cc:511] Master 127.10.191.254:43645 requested a full tablet report, sending...
I20260620 23:13:56.748958 23376 ts_manager.cc:194] Registered new tserver with Master: b20faef2484548919509bf90c9ab7eb5 (127.10.191.194:43103)
I20260620 23:13:56.750082 23376 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.194@KRBTEST.COM'} at 127.10.191.194:39957
WARNING: no policy specified for kudu/127.10.191.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab.
Entry for principal kudu/127.10.191.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab.
I20260620 23:13:56.783233 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.191.212:48067/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.195:0
--local_ip_for_outbound_sockets=127.10.191.195
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:43645
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:33001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:13:57.102891 23589 heartbeater.cc:503] Master 127.10.191.254:43645 was elected leader, sending a full tablet report...
W20260620 23:13:57.107620 23738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:57.107859 23738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:57.108006 23738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:57.115058 23738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:57.115146 23738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:57.115250 23738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.195
I20260620 23:13:57.124776 23738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:33001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.10.191.212:48067/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:43645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.195
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:13:57.125946 23738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:57.127250 23738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:57.140825 23743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:57.140825 23744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:57.142175 23746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:57.142198 23738 server_base.cc:1034] running on GCE node
I20260620 23:13:57.142974 23738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:57.143869 23738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:57.145069 23738 hybrid_clock.cc:648] HybridClock initialized: now 1781997237145038 us; error 36 us; skew 500 ppm
I20260620 23:13:57.145344 23738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:13:57 dist-test-slave-jkbq krb5kdc[22900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997237, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:57.148531 23738 init.cc:377] Logged in from keytab as kudu/127.10.191.195@KRBTEST.COM (short username kudu)
I20260620 23:13:57.149991 23738 webserver.cc:460] Webserver started at http://127.10.191.195:45047/ using document root <none> and password file <none>
I20260620 23:13:57.150686 23738 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:57.150831 23738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:13:57.151170 23738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:13:57 dist-test-slave-jkbq krb5kdc[22900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997237, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for HTTP/127.10.191.212@KRBTEST.COM
I20260620 23:13:57.167249 23738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-2/data/instance:
uuid: "cb6c4ec16b694023b23d938b70a44872"
format_stamp: "Formatted at 2026-06-20 23:13:57 on dist-test-slave-jkbq"
server_key: "7718789455d9de1045e8f37f5afc0597"
server_key_iv: "9b57f84100029423a63eb0fced0c8a19"
server_key_version: "kuduclusterkey@0"
I20260620 23:13:57.168056 23738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance:
uuid: "cb6c4ec16b694023b23d938b70a44872"
format_stamp: "Formatted at 2026-06-20 23:13:57 on dist-test-slave-jkbq"
server_key: "7718789455d9de1045e8f37f5afc0597"
server_key_iv: "9b57f84100029423a63eb0fced0c8a19"
server_key_version: "kuduclusterkey@0"
I20260620 23:13:57.173396 23738 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:13:57.189029 23753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:57.190532 23738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:13:57.190717 23738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-2/wal
uuid: "cb6c4ec16b694023b23d938b70a44872"
format_stamp: "Formatted at 2026-06-20 23:13:57 on dist-test-slave-jkbq"
server_key: "7718789455d9de1045e8f37f5afc0597"
server_key_iv: "9b57f84100029423a63eb0fced0c8a19"
server_key_version: "kuduclusterkey@0"
I20260620 23:13:57.191035 23738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:57.214726 23738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:57.218705 23738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:57.219053 23738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:57.219946 23738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:13:57.221233 23738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:57.221338 23738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:57.221477 23738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:57.221551 23738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:57.254164 23738 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.195:43739
I20260620 23:13:57.254204 23865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.195:43739 every 8 connection(s)
I20260620 23:13:57.255514 23738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
I20260620 23:13:57.264938 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 23738
I20260620 23:13:57.265127 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance
I20260620 23:13:57.265558 11007 external_mini_cluster.cc:1428] Setting key 5d3252be7ff3f43a6fc2d95570d62fbd
Jun 20 23:13:57 dist-test-slave-jkbq krb5kdc[22900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997237, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:13:57.274464 23868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:57.258020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:44199 (local address 127.10.191.254:43645)
0620 23:13:57.258509 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:57.258513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:57.259547 (+ 1034us) server_negotiation.cc:407] Connection header received
0620 23:13:57.261289 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:57.261294 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:57.261387 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:57.261506 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:57.262893 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:57.263685 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:57.265280 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:57.265742 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:57.265867 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:57.269228 (+ 3361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:57.269233 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:57.269236 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:57.269266 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:57.271942 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:57.272592 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:57.272600 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:57.272603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:57.272751 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:57.273129 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:57.273132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:57.273134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:57.273392 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:57.273503 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:57.274063 (+ 560us) server_negotiation.cc:299] Negotiation successful
0620 23:13:57.274220 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":219,"threads_started":1}
I20260620 23:13:57.275417 23869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:57.258553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43645 (local address 127.10.191.195:44199)
0620 23:13:57.259342 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:57.259409 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:57.261029 (+ 1620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:57.261737 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:57.261746 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:57.262159 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:57.262710 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:57.262726 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:57.263838 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:57.263843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:57.265140 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:57.265151 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:57.265966 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:57.265973 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:57.266186 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:57.266852 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:57.266921 (+ 69us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:57.269053 (+ 2132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:57.272074 (+ 3021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:57.272081 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:57.272092 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:57.272465 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:57.272846 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:57.272850 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:57.272852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:57.273038 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:57.273491 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:57.273496 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:57.273770 (+ 274us) client_negotiation.cc:769] Sending connection context
0620 23:13:57.273980 (+ 210us) client_negotiation.cc:240] Negotiation successful
0620 23:13:57.274332 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":329,"threads_started":1}
I20260620 23:13:57.276762 23866 heartbeater.cc:347] Connected to a master server at 127.10.191.254:43645
I20260620 23:13:57.277076 23866 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:57.277721 23866 heartbeater.cc:511] Master 127.10.191.254:43645 requested a full tablet report, sending...
I20260620 23:13:57.279207 23376 ts_manager.cc:194] Registered new tserver with Master: cb6c4ec16b694023b23d938b70a44872 (127.10.191.195:43739)
I20260620 23:13:57.280231 23376 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.195@KRBTEST.COM'} at 127.10.191.195:44199
I20260620 23:13:57.281217 11007 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:13:57.282848 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 23456
Jun 20 23:13:57 dist-test-slave-jkbq krb5kdc[22900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.193@KRBTEST.COM
I20260620 23:13:57.299157 23871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:57.283728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59666 (local address 127.10.191.193:39305)
0620 23:13:57.284294 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:57.284302 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:57.284330 (+ 28us) server_negotiation.cc:407] Connection header received
0620 23:13:57.284530 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:57.284545 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:57.284728 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:57.284884 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:57.286231 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:57.287444 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:57.288920 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:57.289438 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:57.289580 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:57.293844 (+ 4264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:57.293859 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:57.293872 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:57.293904 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:57.296834 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:57.297459 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:57.297465 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:57.297467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:57.297604 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:57.297905 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:57.297909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:57.297911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:57.298212 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:57.298443 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:57.298630 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 23:13:57.298820 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":256,"threads_started":1}
I20260620 23:13:57.300374 23584 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:13:57.381455 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 23456
I20260620 23:13:57.397143 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 23595
Jun 20 23:13:57 dist-test-slave-jkbq krb5kdc[22900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.194@KRBTEST.COM
I20260620 23:13:57.413074 23873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:57.397989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40340 (local address 127.10.191.194:43103)
0620 23:13:57.398559 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:57.398567 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:57.398592 (+ 25us) server_negotiation.cc:407] Connection header received
0620 23:13:57.398652 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:57.398658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:57.398939 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:57.399086 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:57.400322 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:57.401318 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:57.402647 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:57.403331 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:57.403426 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:57.407720 (+ 4294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:57.407730 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:57.407744 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:57.407781 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:57.410248 (+ 2467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:57.410997 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:57.411004 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:57.411007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:57.411133 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:57.411505 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:57.411519 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:57.411523 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:57.411840 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:57.412130 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:57.412441 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 23:13:57.412717 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":296,"threads_started":1}
I20260620 23:13:57.414276 23727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:13:57.495944 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 23595
I20260620 23:13:57.511194 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 23738
Jun 20 23:13:57 dist-test-slave-jkbq krb5kdc[22900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.195@KRBTEST.COM
I20260620 23:13:57.531385 23875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:57.516429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33758 (local address 127.10.191.195:43739)
0620 23:13:57.517001 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:57.517009 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:57.517036 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:13:57.517095 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:57.517101 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:57.517315 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:57.517495 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:57.518910 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:57.519954 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:57.521282 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:57.521827 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:57.521966 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:57.526106 (+ 4140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:57.526117 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:57.526132 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:57.526168 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:57.528914 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:57.529531 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:57.529537 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:57.529539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:57.529658 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:57.529959 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:57.529963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:57.529965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:57.530272 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:57.530521 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:57.530715 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 23:13:57.531093 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":274,"threads_started":1}
I20260620 23:13:57.532608 23861 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:13:57.613722 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 23738
I20260620 23:13:57.629112 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 23344
I20260620 23:13:57.630431 23406 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:13:57.735798 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 23344
I20260620 23:13:57.758445 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:43645
--webserver_interface=127.10.191.254
--webserver_port=41543
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:33001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:43645
--ranger_config_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.191.212:48067/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.191.212:48067/kms with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:13:58.077504 23878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:58.077795 23878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:58.077884 23878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:58.085114 23878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:13:58.085216 23878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:58.085263 23878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:58.085319 23878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:13:58.085366 23878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:13:58.095391 23878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:33001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.191.212:48067/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:43645
--ranger_config_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:43645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=41543
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:13:58.096560 23878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:58.097824 23878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:58.109515 23883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:58.109453 23884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:58.109525 23886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:58.110384 23878 server_base.cc:1034] running on GCE node
I20260620 23:13:58.111194 23878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:58.112561 23878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:58.113788 23878 hybrid_clock.cc:648] HybridClock initialized: now 1781997238113752 us; error 60 us; skew 500 ppm
I20260620 23:13:58.114127 23878 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:13:58 dist-test-slave-jkbq krb5kdc[22900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997238, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:58.117302 23878 init.cc:377] Logged in from keytab as kudu/127.10.191.254@KRBTEST.COM (short username kudu)
I20260620 23:13:58.118970 23878 webserver.cc:460] Webserver started at http://127.10.191.254:41543/ using document root <none> and password file <none>
I20260620 23:13:58.119614 23878 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:58.119764 23878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 23:13:58 dist-test-slave-jkbq krb5kdc[22900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997238, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for HTTP/127.10.191.212@KRBTEST.COM
I20260620 23:13:58.166584 23878 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 23:13:58.169646 23894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:58.171044 23878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:13:58.171231 23878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "9056d50dd8d542c7b9812cc051b61231"
format_stamp: "Formatted at 2026-06-20 23:13:54 on dist-test-slave-jkbq"
server_key: "a684d13650b957bc111aa31ea30d44f8"
server_key_iv: "a7c461f7bc4147fafd768e5201b24c62"
server_key_version: "kuduclusterkey@0"
I20260620 23:13:58.172178 23878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:58.192903 23878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:58.196693 23878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:58.196982 23878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:58.215680 23878 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:43645
I20260620 23:13:58.215718 23945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:43645 every 8 connection(s)
I20260620 23:13:58.217250 23878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:13:58.221015 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 23878
I20260620 23:13:58.222033 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.191.212:48067/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.193:39305
--local_ip_for_outbound_sockets=127.10.191.193
--tserver_master_addrs=127.10.191.254:43645
--webserver_port=46629
--webserver_interface=127.10.191.193
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:33001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.191.212:48067/kms with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:13:58.225621 23946 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:13:58.230559 23946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9056d50dd8d542c7b9812cc051b61231: Bootstrap starting.
I20260620 23:13:58.249226 23946 log.cc:824] T 00000000000000000000000000000000 P 9056d50dd8d542c7b9812cc051b61231: Log is configured to *not* fsync() on all Append() calls
I20260620 23:13:58.257634 23946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9056d50dd8d542c7b9812cc051b61231: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:13:58.258160 23946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9056d50dd8d542c7b9812cc051b61231: Bootstrap complete.
I20260620 23:13:58.267005 23946 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9056d50dd8d542c7b9812cc051b61231 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9056d50dd8d542c7b9812cc051b61231" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43645 } }
I20260620 23:13:58.267822 23946 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9056d50dd8d542c7b9812cc051b61231 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9056d50dd8d542c7b9812cc051b61231, State: Initialized, Role: FOLLOWER
I20260620 23:13:58.268479 23946 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9056d50dd8d542c7b9812cc051b61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9056d50dd8d542c7b9812cc051b61231" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43645 } }
I20260620 23:13:58.268689 23946 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9056d50dd8d542c7b9812cc051b61231 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:13:58.268805 23946 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9056d50dd8d542c7b9812cc051b61231 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:13:58.268991 23946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9056d50dd8d542c7b9812cc051b61231 [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:13:58.272643 23946 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9056d50dd8d542c7b9812cc051b61231 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9056d50dd8d542c7b9812cc051b61231" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43645 } }
I20260620 23:13:58.273120 23946 leader_election.cc:302] T 00000000000000000000000000000000 P 9056d50dd8d542c7b9812cc051b61231 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9056d50dd8d542c7b9812cc051b61231; no voters:
I20260620 23:13:58.273732 23946 leader_election.cc:288] T 00000000000000000000000000000000 P 9056d50dd8d542c7b9812cc051b61231 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 23:13:58.273862 23950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9056d50dd8d542c7b9812cc051b61231 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 23:13:58.275095 23950 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9056d50dd8d542c7b9812cc051b61231 [term 2 LEADER]: Becoming Leader. State: Replica: 9056d50dd8d542c7b9812cc051b61231, State: Running, Role: LEADER
I20260620 23:13:58.275763 23950 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9056d50dd8d542c7b9812cc051b61231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9056d50dd8d542c7b9812cc051b61231" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43645 } }
I20260620 23:13:58.276103 23946 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9056d50dd8d542c7b9812cc051b61231 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:13:58.278224 23952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9056d50dd8d542c7b9812cc051b61231 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9056d50dd8d542c7b9812cc051b61231. Latest consensus state: current_term: 2 leader_uuid: "9056d50dd8d542c7b9812cc051b61231" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9056d50dd8d542c7b9812cc051b61231" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43645 } } }
I20260620 23:13:58.278703 23952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9056d50dd8d542c7b9812cc051b61231 [sys.catalog]: This master's current role is: LEADER
I20260620 23:13:58.279304 23951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9056d50dd8d542c7b9812cc051b61231 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9056d50dd8d542c7b9812cc051b61231" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9056d50dd8d542c7b9812cc051b61231" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43645 } } }
I20260620 23:13:58.279650 23951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9056d50dd8d542c7b9812cc051b61231 [sys.catalog]: This master's current role is: LEADER
I20260620 23:13:58.287673 23956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:13:58.302749 23956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:13:58.304760 23956 catalog_manager.cc:1259] Loaded cluster ID: 5a830c583eda49afb1b052ee5a21259c
I20260620 23:13:58.304980 23956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:13:58.309116 23956 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:13:58.310539 23956 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9056d50dd8d542c7b9812cc051b61231: Loaded TSK: 0
I20260620 23:13:58.312263 23956 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:13:58.315114 23946 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260620 23:13:58.616355 23948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:58.616811 23948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:58.617002 23948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:58.628571 23948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:58.628778 23948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:58.629021 23948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.193
I20260620 23:13:58.641445 23948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:33001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.191.212:48067/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.193:39305
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.191.193
--webserver_port=46629
--tserver_master_addrs=127.10.191.254:43645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.193
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:13:58.642768 23948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:58.644095 23948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:58.673753 23984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:58.675987 23985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:58.680896 23988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:58.681241 23948 server_base.cc:1034] running on GCE node
I20260620 23:13:58.682089 23948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:58.683239 23948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:58.692929 23948 hybrid_clock.cc:648] HybridClock initialized: now 1781997238692837 us; error 65 us; skew 500 ppm
I20260620 23:13:58.693444 23948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:13:58 dist-test-slave-jkbq krb5kdc[22900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997238, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:58.697956 23948 init.cc:377] Logged in from keytab as kudu/127.10.191.193@KRBTEST.COM (short username kudu)
I20260620 23:13:58.712185 23948 webserver.cc:460] Webserver started at http://127.10.191.193:46629/ using document root <none> and password file <none>
I20260620 23:13:58.713027 23948 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:58.713258 23948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 23:13:58 dist-test-slave-jkbq krb5kdc[22900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997238, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for HTTP/127.10.191.212@KRBTEST.COM
I20260620 23:13:58.734388 23948 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 23:13:58.741683 23995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:58.743391 23948 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.006s
I20260620 23:13:58.743589 23948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-0/wal
uuid: "fc11adde099c47558865f18fbde4be49"
format_stamp: "Formatted at 2026-06-20 23:13:55 on dist-test-slave-jkbq"
server_key: "e9ac9139e39c0e00d9465980f622c3c9"
server_key_iv: "cd61d57db488327cd3a7c0b9ba2762c0"
server_key_version: "kuduclusterkey@0"
I20260620 23:13:58.744752 23948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:13:58.768868 23948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:58.773347 23948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:58.773705 23948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:58.781118 23948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:13:58.813558 23948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:58.813782 23948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 23:13:58.814013 23948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:58.814164 23948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:58.955235 23948 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.193:39305
I20260620 23:13:58.955276 24108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.193:39305 every 8 connection(s)
I20260620 23:13:58.956869 23948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
I20260620 23:13:58.962633 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 23948
I20260620 23:13:58.963651 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.191.212:48067/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.194:43103
--local_ip_for_outbound_sockets=127.10.191.194
--tserver_master_addrs=127.10.191.254:43645
--webserver_port=42327
--webserver_interface=127.10.191.194
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:33001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.191.212:48067/kms with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:13:59 dist-test-slave-jkbq krb5kdc[22900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997238, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:13:59.027072 24111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:58.969019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43645 (local address 127.10.191.193:53575)
0620 23:13:58.998292 (+ 29273us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:58.998364 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:59.001201 (+ 2837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:59.007315 (+ 6114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:59.007332 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:59.008241 (+ 909us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:59.009288 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:59.009319 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:59.011844 (+ 2525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:59.011850 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:59.013302 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:59.013369 (+ 67us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:59.014545 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:59.014553 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:59.014917 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:59.015707 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:59.015733 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:59.017844 (+ 2111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:59.021495 (+ 3651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:59.021504 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:59.021520 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:59.021958 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:59.022538 (+ 580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:59.022542 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:59.022548 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:59.022766 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:59.024431 (+ 1665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:59.024440 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:59.024786 (+ 346us) client_negotiation.cc:769] Sending connection context
0620 23:13:59.025053 (+ 267us) client_negotiation.cc:240] Negotiation successful
0620 23:13:59.025653 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28906,"thread_start_us":36257,"threads_started":1}
I20260620 23:13:59.028815 24112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:58.970182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:53575 (local address 127.10.191.254:43645)
0620 23:13:58.997342 (+ 27160us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:58.997369 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:59.001827 (+ 4458us) server_negotiation.cc:407] Connection header received
0620 23:13:59.002282 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:59.002328 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:59.002907 (+ 579us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:59.003335 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:59.009559 (+ 6224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:59.011626 (+ 2067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:59.013547 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:59.014259 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:59.014502 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:59.018138 (+ 3636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:59.018146 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:59.018161 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:59.018200 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:59.021325 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:59.022134 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:59.022143 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:59.022150 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:59.022368 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:59.022926 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:59.022931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:59.022934 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:59.024254 (+ 1320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:59.024483 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:59.025298 (+ 815us) server_negotiation.cc:299] Negotiation successful
0620 23:13:59.025679 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26815,"thread_start_us":27129,"threads_started":1}
I20260620 23:13:59.032516 24109 heartbeater.cc:347] Connected to a master server at 127.10.191.254:43645
I20260620 23:13:59.032840 24109 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:59.033681 24109 heartbeater.cc:511] Master 127.10.191.254:43645 requested a full tablet report, sending...
W20260620 23:13:59.037204 24109 heartbeater.cc:644] Failed to heartbeat to 127.10.191.254:43645 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 23:13:59.037459 24109 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:59.037760 24109 heartbeater.cc:511] Master 127.10.191.254:43645 requested a full tablet report, sending...
I20260620 23:13:59.038988 24109 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:59.039321 24109 heartbeater.cc:511] Master 127.10.191.254:43645 requested a full tablet report, sending...
W20260620 23:13:59.040361 24109 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 23:13:59.404376 24113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:13:59.404835 24113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:13:59.405004 24113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:13:59.416545 24113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:13:59.416745 24113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:13:59.416940 24113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.194
I20260620 23:13:59.432703 24113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:33001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.191.212:48067/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.194:43103
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.191.194
--webserver_port=42327
--tserver_master_addrs=127.10.191.254:43645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.194
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:13:59.434541 24113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:13:59.436389 24113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:13:59.453907 24129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:13:59.455163 24132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:59.456004 24113 server_base.cc:1034] running on GCE node
W20260620 23:13:59.458243 24128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:13:59.459183 24113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:13:59.468645 24113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:13:59.472123 24113 hybrid_clock.cc:648] HybridClock initialized: now 1781997239472031 us; error 94 us; skew 500 ppm
I20260620 23:13:59.472630 24113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:13:59 dist-test-slave-jkbq krb5kdc[22900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997239, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:59.477584 24113 init.cc:377] Logged in from keytab as kudu/127.10.191.194@KRBTEST.COM (short username kudu)
I20260620 23:13:59.481032 24113 webserver.cc:460] Webserver started at http://127.10.191.194:42327/ using document root <none> and password file <none>
I20260620 23:13:59.482049 24113 fs_manager.cc:359] Metadata directory not provided
I20260620 23:13:59.482220 24113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 23:13:59 dist-test-slave-jkbq krb5kdc[22900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997239, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for HTTP/127.10.191.212@KRBTEST.COM
I20260620 23:13:59.518340 24113 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.002s sys 0.003s
I20260620 23:13:59.522316 24140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:59.524315 24113 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:13:59.524501 24113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-1/wal
uuid: "b20faef2484548919509bf90c9ab7eb5"
format_stamp: "Formatted at 2026-06-20 23:13:56 on dist-test-slave-jkbq"
server_key: "ed78cc40d9e1a7b898de81b60df57804"
server_key_iv: "bba1428df34255738324560538956dab"
server_key_version: "kuduclusterkey@0"
I20260620 23:13:59.525959 24113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Jun 20 23:13:59 dist-test-slave-jkbq krb5kdc[22900](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781997239, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:13:59.552152 24113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:13:59.557659 24113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:13:59.558079 24113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:13:59.559332 24113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:13:59.561124 24113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:13:59.561239 24113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:59.561347 24113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:13:59.561426 24113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:13:59.615865 24113 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.194:43103
I20260620 23:13:59.615931 24252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.194:43103 every 8 connection(s)
I20260620 23:13:59.617810 24113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
I20260620 23:13:59.623549 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 24113
I20260620 23:13:59.624323 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.191.212:48067/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.195:43739
--local_ip_for_outbound_sockets=127.10.191.195
--tserver_master_addrs=127.10.191.254:43645
--webserver_port=45047
--webserver_interface=127.10.191.195
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:33001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.191.212:48067/kms with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:13:59 dist-test-slave-jkbq krb5kdc[22900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997239, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:13:59.645452 24256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:59.621158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43645 (local address 127.10.191.194:47557)
0620 23:13:59.621910 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0620 23:13:59.621960 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 23:13:59.624156 (+ 2196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:13:59.625293 (+ 1137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:13:59.625307 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:13:59.625930 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:13:59.626756 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:59.626783 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:59.628470 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:59.628478 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:59.630784 (+ 2306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:13:59.630801 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:13:59.632273 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:13:59.632280 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:13:59.632526 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:59.633300 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:13:59.633333 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:13:59.635943 (+ 2610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:13:59.640279 (+ 4336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:59.640291 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:59.640312 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:59.640950 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:59.641411 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:13:59.641418 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:13:59.641424 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:13:59.641664 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:13:59.642447 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:13:59.642457 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:13:59.642949 (+ 492us) client_negotiation.cc:769] Sending connection context
0620 23:13:59.643264 (+ 315us) client_negotiation.cc:240] Negotiation successful
0620 23:13:59.643872 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":315,"threads_started":1}
I20260620 23:13:59.646720 24255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:13:59.620652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:47557 (local address 127.10.191.254:43645)
0620 23:13:59.621163 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0620 23:13:59.621169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:13:59.624557 (+ 3388us) server_negotiation.cc:407] Connection header received
0620 23:13:59.624632 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:13:59.624640 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:13:59.624732 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:13:59.624834 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:13:59.627000 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:59.628258 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:59.630969 (+ 2711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:13:59.631756 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:13:59.631872 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:13:59.636182 (+ 4310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:13:59.636191 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:13:59.636199 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:13:59.636245 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:13:59.640105 (+ 3860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:59.641117 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:59.641126 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:59.641132 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:59.641283 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:13:59.641799 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:13:59.641806 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:13:59.641812 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:13:59.642204 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:13:59.642374 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 23:13:59.646026 (+ 3652us) server_negotiation.cc:299] Negotiation successful
0620 23:13:59.646293 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":240,"threads_started":1}
I20260620 23:13:59.649433 24253 heartbeater.cc:347] Connected to a master server at 127.10.191.254:43645
I20260620 23:13:59.649814 24253 heartbeater.cc:464] Registering TS with master...
I20260620 23:13:59.650774 24253 heartbeater.cc:511] Master 127.10.191.254:43645 requested a full tablet report, sending...
I20260620 23:13:59.653167 23911 ts_manager.cc:194] Registered new tserver with Master: b20faef2484548919509bf90c9ab7eb5 (127.10.191.194:43103)
I20260620 23:13:59.655647 23911 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.194@KRBTEST.COM'} at 127.10.191.194:47557
Jun 20 23:13:59 dist-test-slave-jkbq krb5kdc[22900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997239, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for HTTP/127.10.191.212@KRBTEST.COM
I20260620 23:14:00.040549 24109 heartbeater.cc:464] Registering TS with master...
I20260620 23:14:00.040966 24109 heartbeater.cc:511] Master 127.10.191.254:43645 requested a full tablet report, sending...
I20260620 23:14:00.042174 23911 ts_manager.cc:194] Registered new tserver with Master: fc11adde099c47558865f18fbde4be49 (127.10.191.193:39305)
I20260620 23:14:00.043439 23911 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.193@KRBTEST.COM'} at 127.10.191.193:53575
W20260620 23:14:00.062573 24258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:14:00.063019 24258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:14:00.063180 24258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:14:00.075014 24258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:14:00.075220 24258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:14:00.075419 24258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.195
I20260620 23:14:00.089728 24258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:33001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.191.212:48067/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.195:43739
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.191.195
--webserver_port=45047
--tserver_master_addrs=127.10.191.254:43645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.195
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:14:00.091204 24258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:14:00.092540 24258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:14:00.109218 24263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:00.109329 24264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:00.114259 24266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:14:00.114571 24258 server_base.cc:1034] running on GCE node
I20260620 23:14:00.115449 24258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:14:00.118384 24258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:14:00.119642 24258 hybrid_clock.cc:648] HybridClock initialized: now 1781997240119554 us; error 69 us; skew 500 ppm
I20260620 23:14:00.120060 24258 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:14:00 dist-test-slave-jkbq krb5kdc[22900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997240, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:14:00.124070 24258 init.cc:377] Logged in from keytab as kudu/127.10.191.195@KRBTEST.COM (short username kudu)
I20260620 23:14:00.126214 24258 webserver.cc:460] Webserver started at http://127.10.191.195:45047/ using document root <none> and password file <none>
I20260620 23:14:00.127324 24258 fs_manager.cc:359] Metadata directory not provided
I20260620 23:14:00.127509 24258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 23:14:00 dist-test-slave-jkbq krb5kdc[22900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997240, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for HTTP/127.10.191.212@KRBTEST.COM
I20260620 23:14:00.152869 24258 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 23:14:00.156412 24274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:00.158213 24258 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 23:14:00.158385 24258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-2/wal
uuid: "cb6c4ec16b694023b23d938b70a44872"
format_stamp: "Formatted at 2026-06-20 23:13:57 on dist-test-slave-jkbq"
server_key: "7718789455d9de1045e8f37f5afc0597"
server_key_iv: "9b57f84100029423a63eb0fced0c8a19"
server_key_version: "kuduclusterkey@0"
I20260620 23:14:00.159737 24258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:14:00.180331 24258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:14:00.186700 24258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:14:00.187130 24258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:14:00.188459 24258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:14:00.190485 24258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:14:00.190670 24258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 23:14:00.190832 24258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:14:00.190984 24258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:00.303965 24258 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.195:43739
I20260620 23:14:00.304059 24386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.195:43739 every 8 connection(s)
I20260620 23:14:00.305951 24258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
I20260620 23:14:00.311798 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 24258
Jun 20 23:14:00 dist-test-slave-jkbq krb5kdc[22900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997240, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:14:00.357296 24393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:00.314668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:52449 (local address 127.10.191.254:43645)
0620 23:14:00.321106 (+ 6438us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:00.321116 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:00.323030 (+ 1914us) server_negotiation.cc:407] Connection header received
0620 23:14:00.331101 (+ 8071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:00.331109 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:00.331209 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:00.331310 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:00.334160 (+ 2850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:00.335367 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:00.337930 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:00.338686 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:00.338784 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:00.348583 (+ 9799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:00.348590 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:00.348596 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:00.348638 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:00.352703 (+ 4065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:00.353912 (+ 1209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:00.353916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:00.353919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:00.354030 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:00.354975 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:00.354980 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:00.354985 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:00.355342 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:00.356594 (+ 1252us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:00.356708 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 23:14:00.356914 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6268,"thread_start_us":13871,"threads_started":1}
Jun 20 23:14:00 dist-test-slave-jkbq krb5kdc[22900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997239, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for HTTP/127.10.191.212@KRBTEST.COM
I20260620 23:14:00.358383 24391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:00.313797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43645 (local address 127.10.191.195:52449)
0620 23:14:00.322071 (+ 8274us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:00.322121 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:00.327318 (+ 5197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:00.332235 (+ 4917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:00.332248 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:00.333030 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:14:00.333930 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:00.333958 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:00.335627 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:00.335632 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:00.337750 (+ 2118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:00.337767 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:00.339206 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:00.339214 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:00.339522 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:00.341268 (+ 1746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:14:00.341302 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:14:00.348316 (+ 7014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:14:00.352845 (+ 4529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:00.352859 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:00.352880 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:00.353505 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:00.354158 (+ 653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:00.354165 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:00.354171 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:00.354457 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:00.355524 (+ 1067us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:14:00.355534 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:14:00.356014 (+ 480us) client_negotiation.cc:769] Sending connection context
0620 23:14:00.356327 (+ 313us) client_negotiation.cc:240] Negotiation successful
0620 23:14:00.356897 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7954,"thread_start_us":7644,"threads_started":1}
I20260620 23:14:00.360085 24387 heartbeater.cc:347] Connected to a master server at 127.10.191.254:43645
I20260620 23:14:00.360399 24387 heartbeater.cc:464] Registering TS with master...
I20260620 23:14:00.361291 24387 heartbeater.cc:511] Master 127.10.191.254:43645 requested a full tablet report, sending...
I20260620 23:14:00.363049 23911 ts_manager.cc:194] Registered new tserver with Master: cb6c4ec16b694023b23d938b70a44872 (127.10.191.195:43739)
I20260620 23:14:00.364161 23911 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.195@KRBTEST.COM'} at 127.10.191.195:52449
I20260620 23:14:00.367278 24392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:00.315170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55774 (local address 127.10.191.254:43645)
0620 23:14:00.328546 (+ 13376us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:00.328556 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:00.335219 (+ 6663us) server_negotiation.cc:407] Connection header received
0620 23:14:00.335304 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:00.335309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:00.335408 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:00.335525 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:00.337572 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:00.338895 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:00.342073 (+ 3178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:00.343616 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:00.343739 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:00.351626 (+ 7887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:00.351633 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:00.351639 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:00.351690 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:00.355759 (+ 4069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:00.357866 (+ 2107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:00.357870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:00.357873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:00.358004 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:00.358726 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:00.358734 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:00.358737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:00.365169 (+ 6432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:00.365443 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:00.366690 (+ 1247us) server_negotiation.cc:299] Negotiation successful
0620 23:14:00.366944 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13195,"thread_start_us":5658,"threads_started":1}
I20260620 23:14:00.369859 11007 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:14:00.390038 24392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:00.373224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55782 (local address 127.10.191.254:43645)
0620 23:14:00.373589 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:00.373594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:00.373957 (+ 363us) server_negotiation.cc:407] Connection header received
0620 23:14:00.374159 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:00.374164 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:00.374246 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:00.374344 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:00.376425 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:00.377487 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:00.379441 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:00.379865 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:00.380022 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:00.383120 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:00.383126 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:00.383130 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:00.383168 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:00.386597 (+ 3429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:00.387605 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:00.387612 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:00.387617 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:00.387776 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:00.388277 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:00.388282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:00.388285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:00.388670 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:00.388909 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:00.389464 (+ 555us) server_negotiation.cc:299] Negotiation successful
0620 23:14:00.389690 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208}
I20260620 23:14:00.395670 23911 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:14:00.406237 23911 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:14:00.493166 24405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:00.458363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:39305 (local address 127.0.0.1:59668)
0620 23:14:00.472081 (+ 13718us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:00.472110 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:00.472271 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:00.474618 (+ 2347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:00.474628 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:00.474679 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:14:00.476115 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:00.476128 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:00.480920 (+ 4792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:00.480926 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:00.483750 (+ 2824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:00.483765 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:00.492065 (+ 8300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:00.492072 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:00.492313 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:00.492371 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 23:14:00.492512 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 23:14:00.492743 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12925,"thread_start_us":11727,"threads_started":1}
I20260620 23:14:00.494688 24406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:00.459298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59668 (local address 127.10.191.193:39305)
0620 23:14:00.472865 (+ 13567us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:00.472875 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:00.472905 (+ 30us) server_negotiation.cc:407] Connection header received
0620 23:14:00.473031 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:00.473040 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:00.473349 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:00.477315 (+ 3966us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:14:00.477589 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:00.480779 (+ 3190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:00.484822 (+ 4043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:00.491245 (+ 6423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:00.491459 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:00.491685 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:00.494135 (+ 2450us) server_negotiation.cc:299] Negotiation successful
0620 23:14:00.494343 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13336,"thread_start_us":11973,"threads_started":1}
I20260620 23:14:00.496711 24403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:00.455580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.194:43103 (local address 127.0.0.1:40346)
0620 23:14:00.461070 (+ 5490us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:00.461135 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:00.461377 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:00.476722 (+ 15345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:00.476731 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:00.476775 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:14:00.477315 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:00.477329 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:00.482823 (+ 5494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:00.482829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:00.485143 (+ 2314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:00.485159 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:00.495919 (+ 10760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:00.495926 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:00.496109 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:00.496143 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 23:14:00.496214 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 23:14:00.496352 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5111,"thread_start_us":8255,"threads_started":1}
I20260620 23:14:00.498194 24404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:00.456157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40346 (local address 127.10.191.194:43103)
0620 23:14:00.476144 (+ 19987us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:00.476152 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:00.476180 (+ 28us) server_negotiation.cc:407] Connection header received
0620 23:14:00.476267 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:00.476274 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:00.476499 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:00.476698 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:14:00.477591 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:00.481101 (+ 3510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:00.486462 (+ 5361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:00.494240 (+ 7778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:00.497334 (+ 3094us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:00.497544 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:00.497621 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 23:14:00.497812 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19824,"thread_start_us":20038,"threads_started":1}
I20260620 23:14:00.500209 24044 tablet_service.cc:1467] Processing CreateTablet for tablet bd86eda7852547d6ac7a2c502d9d5f4b (DEFAULT_TABLE table=test-table [id=5d34b633906c4bfbbeb8e217d97618ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:14:00.502034 24044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd86eda7852547d6ac7a2c502d9d5f4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:14:00.506783 24188 tablet_service.cc:1467] Processing CreateTablet for tablet bd86eda7852547d6ac7a2c502d9d5f4b (DEFAULT_TABLE table=test-table [id=5d34b633906c4bfbbeb8e217d97618ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:14:00.508364 24188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd86eda7852547d6ac7a2c502d9d5f4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:14:00.519393 24408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:00.471694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33774 (local address 127.10.191.195:43739)
0620 23:14:00.472517 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:00.472527 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:00.479283 (+ 6756us) server_negotiation.cc:407] Connection header received
0620 23:14:00.481329 (+ 2046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:00.481340 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:00.481617 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:00.481792 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:14:00.487297 (+ 5505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:00.489088 (+ 1791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:00.498518 (+ 9430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:00.500132 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:00.500326 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:00.500545 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:00.518779 (+ 18234us) server_negotiation.cc:299] Negotiation successful
0620 23:14:00.519038 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":477,"threads_started":1}
I20260620 23:14:00.520418 24407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:00.471222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.195:43739 (local address 127.0.0.1:33774)
0620 23:14:00.479159 (+ 7937us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:00.479180 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:00.480537 (+ 1357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:00.482135 (+ 1598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:00.482140 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:00.482169 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:14:00.482601 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:00.482613 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:00.489756 (+ 7143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:00.489762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:00.495518 (+ 5756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:00.495537 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:00.502364 (+ 6827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:00.502375 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:00.502590 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:00.502625 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 23:14:00.519850 (+ 17225us) client_negotiation.cc:240] Negotiation successful
0620 23:14:00.520031 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7771,"mutex_wait_us":42,"thread_start_us":5702,"threads_started":1}
I20260620 23:14:00.524513 24322 tablet_service.cc:1467] Processing CreateTablet for tablet bd86eda7852547d6ac7a2c502d9d5f4b (DEFAULT_TABLE table=test-table [id=5d34b633906c4bfbbeb8e217d97618ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:14:00.526034 24322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd86eda7852547d6ac7a2c502d9d5f4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:14:00.541884 24409 tablet_bootstrap.cc:492] T bd86eda7852547d6ac7a2c502d9d5f4b P b20faef2484548919509bf90c9ab7eb5: Bootstrap starting.
I20260620 23:14:00.545763 24409 tablet_bootstrap.cc:654] T bd86eda7852547d6ac7a2c502d9d5f4b P b20faef2484548919509bf90c9ab7eb5: Neither blocks nor log segments found. Creating new log.
I20260620 23:14:00.547070 24409 log.cc:824] T bd86eda7852547d6ac7a2c502d9d5f4b P b20faef2484548919509bf90c9ab7eb5: Log is configured to *not* fsync() on all Append() calls
I20260620 23:14:00.547322 24411 tablet_bootstrap.cc:492] T bd86eda7852547d6ac7a2c502d9d5f4b P fc11adde099c47558865f18fbde4be49: Bootstrap starting.
I20260620 23:14:00.550346 24410 tablet_bootstrap.cc:492] T bd86eda7852547d6ac7a2c502d9d5f4b P cb6c4ec16b694023b23d938b70a44872: Bootstrap starting.
I20260620 23:14:00.551435 24411 tablet_bootstrap.cc:654] T bd86eda7852547d6ac7a2c502d9d5f4b P fc11adde099c47558865f18fbde4be49: Neither blocks nor log segments found. Creating new log.
I20260620 23:14:00.553001 24411 log.cc:824] T bd86eda7852547d6ac7a2c502d9d5f4b P fc11adde099c47558865f18fbde4be49: Log is configured to *not* fsync() on all Append() calls
I20260620 23:14:00.554041 24410 tablet_bootstrap.cc:654] T bd86eda7852547d6ac7a2c502d9d5f4b P cb6c4ec16b694023b23d938b70a44872: Neither blocks nor log segments found. Creating new log.
I20260620 23:14:00.555346 24410 log.cc:824] T bd86eda7852547d6ac7a2c502d9d5f4b P cb6c4ec16b694023b23d938b70a44872: Log is configured to *not* fsync() on all Append() calls
I20260620 23:14:00.555420 24409 tablet_bootstrap.cc:492] T bd86eda7852547d6ac7a2c502d9d5f4b P b20faef2484548919509bf90c9ab7eb5: No bootstrap required, opened a new log
I20260620 23:14:00.555864 24409 ts_tablet_manager.cc:1397] T bd86eda7852547d6ac7a2c502d9d5f4b P b20faef2484548919509bf90c9ab7eb5: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.003s
I20260620 23:14:00.564422 24411 tablet_bootstrap.cc:492] T bd86eda7852547d6ac7a2c502d9d5f4b P fc11adde099c47558865f18fbde4be49: No bootstrap required, opened a new log
I20260620 23:14:00.564921 24411 ts_tablet_manager.cc:1397] T bd86eda7852547d6ac7a2c502d9d5f4b P fc11adde099c47558865f18fbde4be49: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.003s
I20260620 23:14:00.571655 24409 raft_consensus.cc:348] T bd86eda7852547d6ac7a2c502d9d5f4b P b20faef2484548919509bf90c9ab7eb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20faef2484548919509bf90c9ab7eb5" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 43103 } } peers { permanent_uuid: "fc11adde099c47558865f18fbde4be49" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39305 } } peers { permanent_uuid: "cb6c4ec16b694023b23d938b70a44872" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 43739 } }
I20260620 23:14:00.572328 24409 raft_consensus.cc:374] T bd86eda7852547d6ac7a2c502d9d5f4b P b20faef2484548919509bf90c9ab7eb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:14:00.572489 24409 raft_consensus.cc:729] T bd86eda7852547d6ac7a2c502d9d5f4b P b20faef2484548919509bf90c9ab7eb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b20faef2484548919509bf90c9ab7eb5, State: Initialized, Role: FOLLOWER
I20260620 23:14:00.573364 24409 consensus_queue.cc:260] T bd86eda7852547d6ac7a2c502d9d5f4b P b20faef2484548919509bf90c9ab7eb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20faef2484548919509bf90c9ab7eb5" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 43103 } } peers { permanent_uuid: "fc11adde099c47558865f18fbde4be49" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39305 } } peers { permanent_uuid: "cb6c4ec16b694023b23d938b70a44872" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 43739 } }
I20260620 23:14:00.580291 24410 tablet_bootstrap.cc:492] T bd86eda7852547d6ac7a2c502d9d5f4b P cb6c4ec16b694023b23d938b70a44872: No bootstrap required, opened a new log
I20260620 23:14:00.580633 24410 ts_tablet_manager.cc:1397] T bd86eda7852547d6ac7a2c502d9d5f4b P cb6c4ec16b694023b23d938b70a44872: Time spent bootstrapping tablet: real 0.031s user 0.012s sys 0.000s
I20260620 23:14:00.581761 24409 ts_tablet_manager.cc:1428] T bd86eda7852547d6ac7a2c502d9d5f4b P b20faef2484548919509bf90c9ab7eb5: Time spent starting tablet: real 0.026s user 0.016s sys 0.004s
I20260620 23:14:00.582340 24253 heartbeater.cc:503] Master 127.10.191.254:43645 was elected leader, sending a full tablet report...
I20260620 23:14:00.587927 24411 raft_consensus.cc:348] T bd86eda7852547d6ac7a2c502d9d5f4b P fc11adde099c47558865f18fbde4be49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20faef2484548919509bf90c9ab7eb5" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 43103 } } peers { permanent_uuid: "fc11adde099c47558865f18fbde4be49" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39305 } } peers { permanent_uuid: "cb6c4ec16b694023b23d938b70a44872" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 43739 } }
I20260620 23:14:00.588497 24411 raft_consensus.cc:374] T bd86eda7852547d6ac7a2c502d9d5f4b P fc11adde099c47558865f18fbde4be49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:14:00.588639 24411 raft_consensus.cc:729] T bd86eda7852547d6ac7a2c502d9d5f4b P fc11adde099c47558865f18fbde4be49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc11adde099c47558865f18fbde4be49, State: Initialized, Role: FOLLOWER
I20260620 23:14:00.589468 24411 consensus_queue.cc:260] T bd86eda7852547d6ac7a2c502d9d5f4b P fc11adde099c47558865f18fbde4be49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20faef2484548919509bf90c9ab7eb5" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 43103 } } peers { permanent_uuid: "fc11adde099c47558865f18fbde4be49" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39305 } } peers { permanent_uuid: "cb6c4ec16b694023b23d938b70a44872" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 43739 } }
I20260620 23:14:00.591593 24109 heartbeater.cc:503] Master 127.10.191.254:43645 was elected leader, sending a full tablet report...
I20260620 23:14:00.596153 24411 ts_tablet_manager.cc:1428] T bd86eda7852547d6ac7a2c502d9d5f4b P fc11adde099c47558865f18fbde4be49: Time spent starting tablet: real 0.031s user 0.012s sys 0.004s
I20260620 23:14:00.596776 24410 raft_consensus.cc:348] T bd86eda7852547d6ac7a2c502d9d5f4b P cb6c4ec16b694023b23d938b70a44872 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20faef2484548919509bf90c9ab7eb5" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 43103 } } peers { permanent_uuid: "fc11adde099c47558865f18fbde4be49" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39305 } } peers { permanent_uuid: "cb6c4ec16b694023b23d938b70a44872" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 43739 } }
I20260620 23:14:00.597301 24410 raft_consensus.cc:374] T bd86eda7852547d6ac7a2c502d9d5f4b P cb6c4ec16b694023b23d938b70a44872 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:14:00.597466 24410 raft_consensus.cc:729] T bd86eda7852547d6ac7a2c502d9d5f4b P cb6c4ec16b694023b23d938b70a44872 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb6c4ec16b694023b23d938b70a44872, State: Initialized, Role: FOLLOWER
I20260620 23:14:00.598287 24410 consensus_queue.cc:260] T bd86eda7852547d6ac7a2c502d9d5f4b P cb6c4ec16b694023b23d938b70a44872 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20faef2484548919509bf90c9ab7eb5" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 43103 } } peers { permanent_uuid: "fc11adde099c47558865f18fbde4be49" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39305 } } peers { permanent_uuid: "cb6c4ec16b694023b23d938b70a44872" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 43739 } }
I20260620 23:14:00.599511 24387 heartbeater.cc:503] Master 127.10.191.254:43645 was elected leader, sending a full tablet report...
I20260620 23:14:00.600116 24410 ts_tablet_manager.cc:1428] T bd86eda7852547d6ac7a2c502d9d5f4b P cb6c4ec16b694023b23d938b70a44872: Time spent starting tablet: real 0.019s user 0.014s sys 0.001s
W20260620 23:14:00.620062 24254 tablet.cc:2369] T bd86eda7852547d6ac7a2c502d9d5f4b P b20faef2484548919509bf90c9ab7eb5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:14:00.670740 24416 raft_consensus.cc:482] T bd86eda7852547d6ac7a2c502d9d5f4b P fc11adde099c47558865f18fbde4be49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:14:00.671135 24416 raft_consensus.cc:504] T bd86eda7852547d6ac7a2c502d9d5f4b P fc11adde099c47558865f18fbde4be49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20faef2484548919509bf90c9ab7eb5" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 43103 } } peers { permanent_uuid: "fc11adde099c47558865f18fbde4be49" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39305 } } peers { permanent_uuid: "cb6c4ec16b694023b23d938b70a44872" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 43739 } }
I20260620 23:14:00.671690 24415 raft_consensus.cc:482] T bd86eda7852547d6ac7a2c502d9d5f4b P b20faef2484548919509bf90c9ab7eb5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:14:00.671960 24415 raft_consensus.cc:504] T bd86eda7852547d6ac7a2c502d9d5f4b P b20faef2484548919509bf90c9ab7eb5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20faef2484548919509bf90c9ab7eb5" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 43103 } } peers { permanent_uuid: "fc11adde099c47558865f18fbde4be49" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39305 } } peers { permanent_uuid: "cb6c4ec16b694023b23d938b70a44872" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 43739 } }
I20260620 23:14:00.673074 24416 leader_election.cc:288] T bd86eda7852547d6ac7a2c502d9d5f4b P fc11adde099c47558865f18fbde4be49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b20faef2484548919509bf90c9ab7eb5 (127.10.191.194:43103), cb6c4ec16b694023b23d938b70a44872 (127.10.191.195:43739)
I20260620 23:14:00.673655 24415 leader_election.cc:288] T bd86eda7852547d6ac7a2c502d9d5f4b P b20faef2484548919509bf90c9ab7eb5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc11adde099c47558865f18fbde4be49 (127.10.191.193:39305), cb6c4ec16b694023b23d938b70a44872 (127.10.191.195:43739)
I20260620 23:14:00.683693 24408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:00.673506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:36057 (local address 127.10.191.195:43739)
0620 23:14:00.673706 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:00.673714 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:00.675294 (+ 1580us) server_negotiation.cc:407] Connection header received
0620 23:14:00.675583 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:00.675591 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:00.675671 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:00.675760 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:14:00.676735 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:00.678164 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:00.681182 (+ 3018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:00.682377 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:00.682513 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:00.682584 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:00.683218 (+ 634us) server_negotiation.cc:299] Negotiation successful
0620 23:14:00.683375 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":64}
I20260620 23:14:00.684331 24404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:00.673704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:35599 (local address 127.10.191.194:43103)
0620 23:14:00.673957 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:00.673966 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:00.674764 (+ 798us) server_negotiation.cc:407] Connection header received
0620 23:14:00.675117 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:00.675124 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:00.675222 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:00.675323 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:14:00.676181 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:00.678079 (+ 1898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:00.681198 (+ 3119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:00.682565 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:00.682686 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:00.682763 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:00.683924 (+ 1161us) server_negotiation.cc:299] Negotiation successful
0620 23:14:00.684064 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 23:14:00.684479 24427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:00.674176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:39305 (local address 127.10.191.194:40147)
0620 23:14:00.675560 (+ 1384us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:00.675577 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:00.675740 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:00.676330 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:00.676334 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:00.676359 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:14:00.677497 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:00.677515 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:00.679803 (+ 2288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:00.679808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:00.681943 (+ 2135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:00.681962 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:00.683627 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:00.683633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:00.683801 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:00.683839 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 23:14:00.683945 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 23:14:00.684085 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1268,"thread_start_us":1066,"threads_started":1}
I20260620 23:14:00.684633 24426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:00.673531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.195:43739 (local address 127.10.191.193:36057)
0620 23:14:00.675182 (+ 1651us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:00.675203 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:00.675407 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:00.676021 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:00.676025 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:00.676050 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:14:00.676544 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:00.676555 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:00.678311 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:00.678315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:00.681006 (+ 2691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:00.681021 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:00.682777 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:00.682786 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:00.683031 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:00.683068 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 23:14:00.684089 (+ 1021us) client_negotiation.cc:240] Negotiation successful
0620 23:14:00.684221 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1316,"mutex_wait_us":214,"thread_start_us":1095,"threads_started":1}
I20260620 23:14:00.685075 24406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:00.674900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:40147 (local address 127.10.191.193:39305)
0620 23:14:00.675097 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:00.675105 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:00.675648 (+ 543us) server_negotiation.cc:407] Connection header received
0620 23:14:00.675885 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:00.675892 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:00.675965 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:00.676043 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:14:00.677957 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:00.679617 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:00.682124 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:00.683428 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:00.683571 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:00.683631 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:00.684677 (+ 1046us) server_negotiation.cc:299] Negotiation successful
0620 23:14:00.684817 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":82,"server-negotiator.queue_time_us":33}
I20260620 23:14:00.685391 24425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:00.673304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.194:43103 (local address 127.10.191.193:35599)
0620 23:14:00.674646 (+ 1342us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:00.674669 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:00.674902 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:00.675532 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:00.675538 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:00.675566 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:14:00.676017 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:00.676031 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:00.678240 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:00.678245 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:00.681009 (+ 2764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:00.681021 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:00.682996 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:00.683002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:00.683745 (+ 743us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:00.683785 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 23:14:00.684840 (+ 1055us) client_negotiation.cc:240] Negotiation successful
0620 23:14:00.684969 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1166,"mutex_wait_us":208,"thread_start_us":946,"threads_started":1}
I20260620 23:14:00.685608 24428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:00.675660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.195:43739 (local address 127.10.191.194:58549)
0620 23:14:00.677991 (+ 2331us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:00.678011 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:00.678134 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:00.678714 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:00.678719 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:00.678743 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:14:00.679224 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:00.679236 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:00.680910 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:00.680915 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:00.683042 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:00.683056 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:00.684819 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:00.684825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:00.684988 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:00.685019 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 23:14:00.685114 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 23:14:00.685234 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2217,"thread_start_us":2044,"threads_started":1}
I20260620 23:14:00.685683 24342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd86eda7852547d6ac7a2c502d9d5f4b" candidate_uuid: "fc11adde099c47558865f18fbde4be49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb6c4ec16b694023b23d938b70a44872" is_pre_election: true
I20260620 23:14:00.686197 24429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:00.676044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:58549 (local address 127.10.191.195:43739)
0620 23:14:00.677043 (+ 999us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:00.677049 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:00.678234 (+ 1185us) server_negotiation.cc:407] Connection header received
0620 23:14:00.678296 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:00.678302 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:00.678375 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:00.678466 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:14:00.679571 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:00.680749 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:00.683242 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:00.684471 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:00.684626 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:00.684692 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:00.685651 (+ 959us) server_negotiation.cc:299] Negotiation successful
0620 23:14:00.685813 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":440,"threads_started":1}
I20260620 23:14:00.686319 24342 raft_consensus.cc:2454] T bd86eda7852547d6ac7a2c502d9d5f4b P cb6c4ec16b694023b23d938b70a44872 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc11adde099c47558865f18fbde4be49 in term 0.
I20260620 23:14:00.686350 24208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd86eda7852547d6ac7a2c502d9d5f4b" candidate_uuid: "fc11adde099c47558865f18fbde4be49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b20faef2484548919509bf90c9ab7eb5" is_pre_election: true
I20260620 23:14:00.686764 24208 raft_consensus.cc:2454] T bd86eda7852547d6ac7a2c502d9d5f4b P b20faef2484548919509bf90c9ab7eb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc11adde099c47558865f18fbde4be49 in term 0.
I20260620 23:14:00.686964 24341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd86eda7852547d6ac7a2c502d9d5f4b" candidate_uuid: "b20faef2484548919509bf90c9ab7eb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb6c4ec16b694023b23d938b70a44872" is_pre_election: true
I20260620 23:14:00.687237 24341 raft_consensus.cc:2454] T bd86eda7852547d6ac7a2c502d9d5f4b P cb6c4ec16b694023b23d938b70a44872 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b20faef2484548919509bf90c9ab7eb5 in term 0.
I20260620 23:14:00.687260 23999 leader_election.cc:302] T bd86eda7852547d6ac7a2c502d9d5f4b P fc11adde099c47558865f18fbde4be49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb6c4ec16b694023b23d938b70a44872, fc11adde099c47558865f18fbde4be49; no voters:
I20260620 23:14:00.687516 24064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd86eda7852547d6ac7a2c502d9d5f4b" candidate_uuid: "b20faef2484548919509bf90c9ab7eb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc11adde099c47558865f18fbde4be49" is_pre_election: true
I20260620 23:14:00.687783 24144 leader_election.cc:302] T bd86eda7852547d6ac7a2c502d9d5f4b P b20faef2484548919509bf90c9ab7eb5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b20faef2484548919509bf90c9ab7eb5, cb6c4ec16b694023b23d938b70a44872; no voters:
I20260620 23:14:00.687944 24064 raft_consensus.cc:2454] T bd86eda7852547d6ac7a2c502d9d5f4b P fc11adde099c47558865f18fbde4be49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b20faef2484548919509bf90c9ab7eb5 in term 0.
I20260620 23:14:00.688155 24415 raft_consensus.cc:2781] T bd86eda7852547d6ac7a2c502d9d5f4b P b20faef2484548919509bf90c9ab7eb5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:14:00.688299 24415 raft_consensus.cc:482] T bd86eda7852547d6ac7a2c502d9d5f4b P b20faef2484548919509bf90c9ab7eb5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:14:00.688302 24416 raft_consensus.cc:2781] T bd86eda7852547d6ac7a2c502d9d5f4b P fc11adde099c47558865f18fbde4be49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:14:00.688434 24415 raft_consensus.cc:3037] T bd86eda7852547d6ac7a2c502d9d5f4b P b20faef2484548919509bf90c9ab7eb5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:14:00.688439 24416 raft_consensus.cc:482] T bd86eda7852547d6ac7a2c502d9d5f4b P fc11adde099c47558865f18fbde4be49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:14:00.688529 24416 raft_consensus.cc:3037] T bd86eda7852547d6ac7a2c502d9d5f4b P fc11adde099c47558865f18fbde4be49 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:14:00.691653 24415 raft_consensus.cc:504] T bd86eda7852547d6ac7a2c502d9d5f4b P b20faef2484548919509bf90c9ab7eb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20faef2484548919509bf90c9ab7eb5" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 43103 } } peers { permanent_uuid: "fc11adde099c47558865f18fbde4be49" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39305 } } peers { permanent_uuid: "cb6c4ec16b694023b23d938b70a44872" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 43739 } }
I20260620 23:14:00.691653 24416 raft_consensus.cc:504] T bd86eda7852547d6ac7a2c502d9d5f4b P fc11adde099c47558865f18fbde4be49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20faef2484548919509bf90c9ab7eb5" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 43103 } } peers { permanent_uuid: "fc11adde099c47558865f18fbde4be49" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39305 } } peers { permanent_uuid: "cb6c4ec16b694023b23d938b70a44872" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 43739 } }
I20260620 23:14:00.692304 24416 leader_election.cc:288] T bd86eda7852547d6ac7a2c502d9d5f4b P fc11adde099c47558865f18fbde4be49 [CANDIDATE]: Term 1 election: Requested vote from peers b20faef2484548919509bf90c9ab7eb5 (127.10.191.194:43103), cb6c4ec16b694023b23d938b70a44872 (127.10.191.195:43739)
I20260620 23:14:00.692294 24415 leader_election.cc:288] T bd86eda7852547d6ac7a2c502d9d5f4b P b20faef2484548919509bf90c9ab7eb5 [CANDIDATE]: Term 1 election: Requested vote from peers fc11adde099c47558865f18fbde4be49 (127.10.191.193:39305), cb6c4ec16b694023b23d938b70a44872 (127.10.191.195:43739)
I20260620 23:14:00.692672 24208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd86eda7852547d6ac7a2c502d9d5f4b" candidate_uuid: "fc11adde099c47558865f18fbde4be49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b20faef2484548919509bf90c9ab7eb5"
I20260620 23:14:00.692755 24341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd86eda7852547d6ac7a2c502d9d5f4b" candidate_uuid: "b20faef2484548919509bf90c9ab7eb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb6c4ec16b694023b23d938b70a44872"
I20260620 23:14:00.692955 24208 raft_consensus.cc:2379] T bd86eda7852547d6ac7a2c502d9d5f4b P b20faef2484548919509bf90c9ab7eb5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate fc11adde099c47558865f18fbde4be49 in current term 1: Already voted for candidate b20faef2484548919509bf90c9ab7eb5 in this term.
I20260620 23:14:00.692884 24342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd86eda7852547d6ac7a2c502d9d5f4b" candidate_uuid: "fc11adde099c47558865f18fbde4be49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb6c4ec16b694023b23d938b70a44872"
I20260620 23:14:00.692974 24341 raft_consensus.cc:3037] T bd86eda7852547d6ac7a2c502d9d5f4b P cb6c4ec16b694023b23d938b70a44872 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:14:00.693218 24064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd86eda7852547d6ac7a2c502d9d5f4b" candidate_uuid: "b20faef2484548919509bf90c9ab7eb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc11adde099c47558865f18fbde4be49"
I20260620 23:14:00.693502 24064 raft_consensus.cc:2379] T bd86eda7852547d6ac7a2c502d9d5f4b P fc11adde099c47558865f18fbde4be49 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b20faef2484548919509bf90c9ab7eb5 in current term 1: Already voted for candidate fc11adde099c47558865f18fbde4be49 in this term.
I20260620 23:14:00.693686 23999 leader_election.cc:302] T bd86eda7852547d6ac7a2c502d9d5f4b P fc11adde099c47558865f18fbde4be49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fc11adde099c47558865f18fbde4be49; no voters: b20faef2484548919509bf90c9ab7eb5, cb6c4ec16b694023b23d938b70a44872
I20260620 23:14:00.693989 24416 raft_consensus.cc:2726] T bd86eda7852547d6ac7a2c502d9d5f4b P fc11adde099c47558865f18fbde4be49 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260620 23:14:00.695734 24341 raft_consensus.cc:2454] T bd86eda7852547d6ac7a2c502d9d5f4b P cb6c4ec16b694023b23d938b70a44872 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b20faef2484548919509bf90c9ab7eb5 in term 1.
I20260620 23:14:00.696144 24144 leader_election.cc:302] T bd86eda7852547d6ac7a2c502d9d5f4b P b20faef2484548919509bf90c9ab7eb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b20faef2484548919509bf90c9ab7eb5, cb6c4ec16b694023b23d938b70a44872; no voters: fc11adde099c47558865f18fbde4be49
I20260620 23:14:00.696405 24415 raft_consensus.cc:2781] T bd86eda7852547d6ac7a2c502d9d5f4b P b20faef2484548919509bf90c9ab7eb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:14:00.696862 24415 raft_consensus.cc:686] T bd86eda7852547d6ac7a2c502d9d5f4b P b20faef2484548919509bf90c9ab7eb5 [term 1 LEADER]: Becoming Leader. State: Replica: b20faef2484548919509bf90c9ab7eb5, State: Running, Role: LEADER
I20260620 23:14:00.697290 24415 consensus_queue.cc:237] T bd86eda7852547d6ac7a2c502d9d5f4b P b20faef2484548919509bf90c9ab7eb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20faef2484548919509bf90c9ab7eb5" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 43103 } } peers { permanent_uuid: "fc11adde099c47558865f18fbde4be49" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39305 } } peers { permanent_uuid: "cb6c4ec16b694023b23d938b70a44872" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 43739 } }
I20260620 23:14:00.701774 23910 catalog_manager.cc:5495] T bd86eda7852547d6ac7a2c502d9d5f4b P b20faef2484548919509bf90c9ab7eb5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b20faef2484548919509bf90c9ab7eb5 (127.10.191.194). New cstate: current_term: 1 leader_uuid: "b20faef2484548919509bf90c9ab7eb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20faef2484548919509bf90c9ab7eb5" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 43103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc11adde099c47558865f18fbde4be49" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb6c4ec16b694023b23d938b70a44872" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 43739 } health_report { overall_health: UNKNOWN } } }
W20260620 23:14:00.761478 24110 tablet.cc:2369] T bd86eda7852547d6ac7a2c502d9d5f4b P fc11adde099c47558865f18fbde4be49: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:14:00.770001 24404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:00.764448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40352 (local address 127.10.191.194:43103)
0620 23:14:00.764614 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:00.764618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:00.764634 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:14:00.764715 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:00.764718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:00.764789 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:00.764909 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:14:00.765344 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:00.766132 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:00.767531 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:00.768012 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:00.768159 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:00.768761 (+ 602us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:14:00.769400 (+ 639us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:14:00.769573 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:00.769626 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 23:14:00.769704 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 23:14:00.778223 24064 raft_consensus.cc:1261] T bd86eda7852547d6ac7a2c502d9d5f4b P fc11adde099c47558865f18fbde4be49 [term 1 FOLLOWER]: Refusing update from remote peer b20faef2484548919509bf90c9ab7eb5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:14:00.778234 24341 raft_consensus.cc:1261] T bd86eda7852547d6ac7a2c502d9d5f4b P cb6c4ec16b694023b23d938b70a44872 [term 1 FOLLOWER]: Refusing update from remote peer b20faef2484548919509bf90c9ab7eb5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:14:00.779126 24430 consensus_queue.cc:1035] T bd86eda7852547d6ac7a2c502d9d5f4b P b20faef2484548919509bf90c9ab7eb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb6c4ec16b694023b23d938b70a44872" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 43739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:14:00.779592 24415 consensus_queue.cc:1035] T bd86eda7852547d6ac7a2c502d9d5f4b P b20faef2484548919509bf90c9ab7eb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc11adde099c47558865f18fbde4be49" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 39305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:14:00.790589 24433 mvcc.cc:205] Tried to move back new op lower bound from 7299060698217709568 to 7299060697899491328. Current Snapshot: MvccSnapshot[applied={T|T < 7299060698217709568}]
I20260620 23:14:00.806201 23911 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:14:00.806500 23911 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:14:00.809767 23911 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9056d50dd8d542c7b9812cc051b61231: Sending DeleteTablet for 3 replicas of tablet bd86eda7852547d6ac7a2c502d9d5f4b
I20260620 23:14:00.810804 24188 tablet_service.cc:1514] Processing DeleteTablet for tablet bd86eda7852547d6ac7a2c502d9d5f4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:14:00 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:40346
I20260620 23:14:00.810961 24322 tablet_service.cc:1514] Processing DeleteTablet for tablet bd86eda7852547d6ac7a2c502d9d5f4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:14:00 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:33774
I20260620 23:14:00.810941 24044 tablet_service.cc:1514] Processing DeleteTablet for tablet bd86eda7852547d6ac7a2c502d9d5f4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:14:00 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:59668
I20260620 23:14:00.819989 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 23948
I20260620 23:14:00.826556 24445 tablet_replica.cc:330] T bd86eda7852547d6ac7a2c502d9d5f4b P fc11adde099c47558865f18fbde4be49: stopping tablet replica
I20260620 23:14:00.827041 24445 raft_consensus.cc:2229] T bd86eda7852547d6ac7a2c502d9d5f4b P fc11adde099c47558865f18fbde4be49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:14:00.827585 24445 raft_consensus.cc:2258] T bd86eda7852547d6ac7a2c502d9d5f4b P fc11adde099c47558865f18fbde4be49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:14:00.832118 24443 tablet_replica.cc:330] T bd86eda7852547d6ac7a2c502d9d5f4b P b20faef2484548919509bf90c9ab7eb5: stopping tablet replica
I20260620 23:14:00.832670 24443 raft_consensus.cc:2229] T bd86eda7852547d6ac7a2c502d9d5f4b P b20faef2484548919509bf90c9ab7eb5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:14:00.832921 24444 tablet_replica.cc:330] T bd86eda7852547d6ac7a2c502d9d5f4b P cb6c4ec16b694023b23d938b70a44872: stopping tablet replica
I20260620 23:14:00.833376 24444 raft_consensus.cc:2229] T bd86eda7852547d6ac7a2c502d9d5f4b P cb6c4ec16b694023b23d938b70a44872 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:14:00.833529 24443 raft_consensus.cc:2258] T bd86eda7852547d6ac7a2c502d9d5f4b P b20faef2484548919509bf90c9ab7eb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:14:00.833770 24445 ts_tablet_manager.cc:1905] T bd86eda7852547d6ac7a2c502d9d5f4b P fc11adde099c47558865f18fbde4be49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:14:00.833961 24444 raft_consensus.cc:2258] T bd86eda7852547d6ac7a2c502d9d5f4b P cb6c4ec16b694023b23d938b70a44872 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:14:00.835876 24444 ts_tablet_manager.cc:1905] T bd86eda7852547d6ac7a2c502d9d5f4b P cb6c4ec16b694023b23d938b70a44872: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:14:00.841185 24445 ts_tablet_manager.cc:1918] T bd86eda7852547d6ac7a2c502d9d5f4b P fc11adde099c47558865f18fbde4be49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:14:00.841378 24445 log.cc:1196] T bd86eda7852547d6ac7a2c502d9d5f4b P fc11adde099c47558865f18fbde4be49: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-0/wal/wals/bd86eda7852547d6ac7a2c502d9d5f4b
I20260620 23:14:00.842005 24445 ts_tablet_manager.cc:1939] T bd86eda7852547d6ac7a2c502d9d5f4b P fc11adde099c47558865f18fbde4be49: Deleting consensus metadata
I20260620 23:14:00.842893 24444 ts_tablet_manager.cc:1918] T bd86eda7852547d6ac7a2c502d9d5f4b P cb6c4ec16b694023b23d938b70a44872: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:14:00.843132 24444 log.cc:1196] T bd86eda7852547d6ac7a2c502d9d5f4b P cb6c4ec16b694023b23d938b70a44872: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-2/wal/wals/bd86eda7852547d6ac7a2c502d9d5f4b
I20260620 23:14:00.843698 24444 ts_tablet_manager.cc:1939] T bd86eda7852547d6ac7a2c502d9d5f4b P cb6c4ec16b694023b23d938b70a44872: Deleting consensus metadata
I20260620 23:14:00.843982 23898 catalog_manager.cc:4841] TS fc11adde099c47558865f18fbde4be49 (127.10.191.193:39305): tablet bd86eda7852547d6ac7a2c502d9d5f4b (table test-table [id=5d34b633906c4bfbbeb8e217d97618ec]) successfully deleted
I20260620 23:14:00.844528 24443 ts_tablet_manager.cc:1905] T bd86eda7852547d6ac7a2c502d9d5f4b P b20faef2484548919509bf90c9ab7eb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:14:00.845417 23898 catalog_manager.cc:4841] TS cb6c4ec16b694023b23d938b70a44872 (127.10.191.195:43739): tablet bd86eda7852547d6ac7a2c502d9d5f4b (table test-table [id=5d34b633906c4bfbbeb8e217d97618ec]) successfully deleted
I20260620 23:14:00.847420 24406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:00.821863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59678 (local address 127.10.191.193:39305)
0620 23:14:00.822092 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:00.822100 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:00.822124 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:14:00.822198 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:00.822206 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:00.822320 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:00.822457 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:00.824896 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:00.826412 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:00.828915 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:00.829789 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:00.829936 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:00.838391 (+ 8455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:00.838409 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:00.838425 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:00.838474 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:00.843625 (+ 5151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:00.845126 (+ 1501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:00.845137 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:00.845146 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:00.845324 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:00.845723 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:00.845728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:00.845732 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:00.846093 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:00.846395 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:00.846787 (+ 392us) server_negotiation.cc:299] Negotiation successful
0620 23:14:00.847050 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 23:14:00.848098 24104 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:14:00.854928 24443 ts_tablet_manager.cc:1918] T bd86eda7852547d6ac7a2c502d9d5f4b P b20faef2484548919509bf90c9ab7eb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:14:00.855132 24443 log.cc:1196] T bd86eda7852547d6ac7a2c502d9d5f4b P b20faef2484548919509bf90c9ab7eb5: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781997042893763-11007-0/minicluster-data/ts-1/wal/wals/bd86eda7852547d6ac7a2c502d9d5f4b
I20260620 23:14:00.855625 24443 ts_tablet_manager.cc:1939] T bd86eda7852547d6ac7a2c502d9d5f4b P b20faef2484548919509bf90c9ab7eb5: Deleting consensus metadata
I20260620 23:14:00.857080 23895 catalog_manager.cc:4841] TS b20faef2484548919509bf90c9ab7eb5 (127.10.191.194:43103): tablet bd86eda7852547d6ac7a2c502d9d5f4b (table test-table [id=5d34b633906c4bfbbeb8e217d97618ec]) successfully deleted
I20260620 23:14:00.983613 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 23948
I20260620 23:14:01.001804 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 24113
I20260620 23:14:01.016836 24404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:01.002904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40364 (local address 127.10.191.194:43103)
0620 23:14:01.003122 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:01.003130 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:01.003152 (+ 22us) server_negotiation.cc:407] Connection header received
0620 23:14:01.003217 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:01.003227 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:01.003285 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:01.003392 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:01.004653 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:01.005398 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:01.006664 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:01.007159 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:01.007301 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:01.010855 (+ 3554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:01.010923 (+ 68us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:01.010944 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:01.011008 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:01.014197 (+ 3189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:01.015020 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:01.015026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:01.015029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:01.015146 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:01.015533 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:01.015536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:01.015538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:01.015908 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:01.016134 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:01.016444 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 23:14:01.016613 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 23:14:01.017755 24248 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:14:01.131700 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 24113
I20260620 23:14:01.150486 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 24258
I20260620 23:14:01.164288 24429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:01.151325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33790 (local address 127.10.191.195:43739)
0620 23:14:01.151522 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:01.151530 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:01.151551 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:14:01.151768 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:01.151774 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:01.151852 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:01.151959 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:01.153261 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:01.153972 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:01.155405 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:01.155805 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:01.155944 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:01.158524 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:01.158540 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:01.158556 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:01.158605 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:01.161314 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:01.162053 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:01.162066 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:01.162068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:01.162198 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:01.162579 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:01.162586 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:01.162589 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:01.163074 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:01.163359 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:01.163701 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 23:14:01.163975 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 23:14:01.164929 24382 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:14:01.252817 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 24258
I20260620 23:14:01.268914 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 23878
I20260620 23:14:01.270084 23941 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:14:01.383836 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 23878
I20260620 23:14:01.406467 11007 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 23:14:01.741469 11007 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 23:14:01.741664 11007 mini_ranger.cc:66] Stopping Ranger...
I20260620 23:14:02.086990 11007 mini_ranger.cc:68] Stopped Ranger
2026-06-20 23:14:02.087 UTC [22930] LOG: received smart shutdown request
2026-06-20 23:14:02.090 UTC [22930] LOG: background worker "logical replication launcher" (PID 22939) exited with exit code 1
2026-06-20 23:14:02.090 UTC [22934] LOG: shutting down
2026-06-20 23:14:02.241 UTC [22930] LOG: database system is shut down
2026-06-20T23:14:02Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51842 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:14:02 dist-test-slave-jkbq krb5kdc[24458](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:14:02 dist-test-slave-jkbq krb5kdc[24458](info): set up 2 sockets
Jun 20 23:14:02 dist-test-slave-jkbq krb5kdc[24458](info): commencing operation
krb5kdc: starting...
W20260620 23:14:03.467180 11007 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Jun 20 23:14:03 dist-test-slave-jkbq krb5kdc[24458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:14:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:14:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.191.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
I20260620 23:14:03.681394 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:42913
--webserver_interface=127.10.191.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:37381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:42913,127.10.191.253:33525,127.10.191.252:42595
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:14:04.002740 24471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:14:04.003103 24471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:14:04.003262 24471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:14:04.010449 24471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:14:04.010597 24471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:14:04.010707 24471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:14:04.010809 24471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:14:04.010910 24471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:14:04.021013 24471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:37381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:42913,127.10.191.253:33525,127.10.191.252:42595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:42913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:14:04.022722 24471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:14:04.024120 24471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:14:04.035857 24476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:04.035916 24477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:04.035830 24479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:14:04.036891 24471 server_base.cc:1034] running on GCE node
I20260620 23:14:04.037659 24471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:14:04.039052 24471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:14:04.040274 24471 hybrid_clock.cc:648] HybridClock initialized: now 1781997244040227 us; error 53 us; skew 500 ppm
I20260620 23:14:04.040628 24471 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:14:04 dist-test-slave-jkbq krb5kdc[24458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997244, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:14:04.043744 24471 init.cc:377] Logged in from keytab as kudu/127.10.191.254@KRBTEST.COM (short username kudu)
I20260620 23:14:04.045219 24471 webserver.cc:460] Webserver started at http://127.10.191.254:36491/ using document root <none> and password file <none>
I20260620 23:14:04.045894 24471 fs_manager.cc:359] Metadata directory not provided
I20260620 23:14:04.046029 24471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:14:04.046325 24471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:14:04.049223 24471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "6c46bab565ee4f9aa2bc4ff19aab2f19"
format_stamp: "Formatted at 2026-06-20 23:14:04 on dist-test-slave-jkbq"
server_key: "d0240bcc5218b2495f2a7a64e7e9b865"
server_key_iv: "31598ff89eb38ea470028c52c721cd88"
server_key_version: "encryptionkey@0"
I20260620 23:14:04.050067 24471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "6c46bab565ee4f9aa2bc4ff19aab2f19"
format_stamp: "Formatted at 2026-06-20 23:14:04 on dist-test-slave-jkbq"
server_key: "d0240bcc5218b2495f2a7a64e7e9b865"
server_key_iv: "31598ff89eb38ea470028c52c721cd88"
server_key_version: "encryptionkey@0"
I20260620 23:14:04.055024 24471 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260620 23:14:04.058254 24486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:04.059640 24471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:14:04.059811 24471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "6c46bab565ee4f9aa2bc4ff19aab2f19"
format_stamp: "Formatted at 2026-06-20 23:14:04 on dist-test-slave-jkbq"
server_key: "d0240bcc5218b2495f2a7a64e7e9b865"
server_key_iv: "31598ff89eb38ea470028c52c721cd88"
server_key_version: "encryptionkey@0"
I20260620 23:14:04.060017 24471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:14:04.090018 24471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:14:04.094208 24471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:14:04.094460 24471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:14:04.112821 24471 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:42913
I20260620 23:14:04.112857 24537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:42913 every 8 connection(s)
I20260620 23:14:04.114145 24471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:14:04.117787 24538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:14:04.121194 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 24471
I20260620 23:14:04.121433 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:14:04.122082 11007 external_mini_cluster.cc:1428] Setting key fa0e21e6783298637500504ecdc3924f
I20260620 23:14:04.124408 24538 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.191.254" port: 42913 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:14:04 dist-test-slave-jkbq krb5kdc[24458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997244, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.191.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.253@KRBTEST.COM" created.
I20260620 23:14:04.148250 24540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:04.127494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:42913 (local address 127.0.0.1:53106)
0620 23:14:04.128358 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:04.128410 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:04.130900 (+ 2490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:04.131767 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:04.131777 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:04.132610 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:14:04.133114 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:04.133140 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:04.134630 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:04.134639 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:04.136541 (+ 1902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:04.136558 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:04.137473 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:04.137483 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:04.137713 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:04.138407 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:14:04.138452 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:14:04.141071 (+ 2619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:14:04.144390 (+ 3319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:04.144400 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:04.144411 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:04.144810 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:04.145228 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:04.145232 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:04.145236 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:04.145422 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:04.146369 (+ 947us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:14:04.146376 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:14:04.146557 (+ 181us) client_negotiation.cc:769] Sending connection context
0620 23:14:04.146774 (+ 217us) client_negotiation.cc:240] Negotiation successful
0620 23:14:04.147210 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":364,"threads_started":1}
I20260620 23:14:04.148278 24541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:04.127495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53106 (local address 127.10.191.254:42913)
0620 23:14:04.128608 (+ 1113us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:04.128618 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:04.128672 (+ 54us) server_negotiation.cc:407] Connection header received
0620 23:14:04.131261 (+ 2589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:04.131267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:04.131517 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:04.131681 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:04.133311 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:04.134471 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:04.136678 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:04.137220 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:04.137359 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:04.141285 (+ 3926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:04.141293 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:04.141305 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:04.141339 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:04.144237 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:04.144972 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:04.144978 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:04.144980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:04.145111 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:04.145535 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:04.145545 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:04.145547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:04.146193 (+ 646us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:04.146406 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:04.146995 (+ 589us) server_negotiation.cc:299] Negotiation successful
0620 23:14:04.147404 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":73,"server-negotiator.queue_time_us":886,"spinlock_wait_cycles":16256,"thread_start_us":676,"threads_started":1}
I20260620 23:14:04.151175 24538 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33525 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:14:04.152117 24540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:04.151748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:33525 (local address 127.0.0.1:37530)
0620 23:14:04.151901 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:04.152004 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.253:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 23:14:04.152427 24489 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.191.253:33525: connect: Connection refused (error 111)
I20260620 23:14:04.153924 24540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:04.153604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:33525 (local address 127.0.0.1:37546)
0620 23:14:04.153741 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:04.153809 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.253:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 23:14:04.154150 24538 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.191.253:33525: Network error: Client connection negotiation failed: client connection to 127.10.191.253:33525: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.253.keytab.
Entry for principal kudu/127.10.191.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.253.keytab.
I20260620 23:14:04.174088 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.253:33525
--webserver_interface=127.10.191.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.253.keytab
--principal=kudu/127.10.191.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:37381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:42913,127.10.191.253:33525,127.10.191.252:42595
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:14:04.203509 24538 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33525 } attempt: 1
I20260620 23:14:04.204725 24540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:04.204250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:33525 (local address 127.0.0.1:37552)
0620 23:14:04.204469 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:04.204582 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.253:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 23:14:04.205675 24540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:04.205336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:33525 (local address 127.0.0.1:37566)
0620 23:14:04.205473 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:04.205547 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.253:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 23:14:04.205863 24538 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.191.253:33525: Network error: Client connection negotiation failed: client connection to 127.10.191.253:33525: connect: Connection refused (error 111)
I20260620 23:14:04.274147 24538 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33525 } attempt: 2
I20260620 23:14:04.275393 24540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:04.274903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:33525 (local address 127.0.0.1:37568)
0620 23:14:04.275156 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:04.275253 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.253:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 23:14:04.276451 24540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:04.276117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:33525 (local address 127.0.0.1:37574)
0620 23:14:04.276242 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:04.276322 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.253:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 23:14:04.276748 24538 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.191.253:33525: Network error: Client connection negotiation failed: client connection to 127.10.191.253:33525: connect: Connection refused (error 111)
I20260620 23:14:04.368005 24538 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33525 } attempt: 3
I20260620 23:14:04.369134 24540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:04.368707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:33525 (local address 127.0.0.1:37576)
0620 23:14:04.368906 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:04.369001 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.253:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 23:14:04.370172 24540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:04.369786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:33525 (local address 127.0.0.1:37582)
0620 23:14:04.369928 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:04.370037 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.253:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 23:14:04.370342 24538 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.191.253:33525: Network error: Client connection negotiation failed: client connection to 127.10.191.253:33525: connect: Connection refused (error 111)
W20260620 23:14:04.503671 24545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:14:04.503995 24545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:14:04.504226 24545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:14:04.511338 24545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:14:04.511487 24545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:14:04.511574 24545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:14:04.511677 24545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:14:04.511757 24545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:14:04.513589 24538 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33525 } attempt: 4
I20260620 23:14:04.514823 24540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:04.514196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:33525 (local address 127.0.0.1:37598)
0620 23:14:04.514543 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:04.514656 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.253:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
I20260620 23:14:04.515931 24540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:04.515513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:33525 (local address 127.0.0.1:37600)
0620 23:14:04.515664 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:04.515774 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.253:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 23:14:04.516130 24538 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.191.253:33525: Network error: Client connection negotiation failed: client connection to 127.10.191.253:33525: connect: Connection refused (error 111)
I20260620 23:14:04.522250 24545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:37381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:42913,127.10.191.253:33525,127.10.191.252:42595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.253.keytab
--principal=kudu/127.10.191.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.253:33525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.191.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:14:04.523778 24545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:14:04.525116 24545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:14:04.536814 24550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:04.536911 24553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:04.536935 24551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:14:04.538069 24545 server_base.cc:1034] running on GCE node
I20260620 23:14:04.538827 24545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:14:04.540253 24545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:14:04.541446 24545 hybrid_clock.cc:648] HybridClock initialized: now 1781997244541405 us; error 33 us; skew 500 ppm
I20260620 23:14:04.541806 24545 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:14:04 dist-test-slave-jkbq krb5kdc[24458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997244, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:14:04.545008 24545 init.cc:377] Logged in from keytab as kudu/127.10.191.253@KRBTEST.COM (short username kudu)
I20260620 23:14:04.546598 24545 webserver.cc:460] Webserver started at http://127.10.191.253:45573/ using document root <none> and password file <none>
I20260620 23:14:04.547345 24545 fs_manager.cc:359] Metadata directory not provided
I20260620 23:14:04.547480 24545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:14:04.547784 24545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:14:04.550757 24545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-1/data/instance:
uuid: "a8ad0769ea444ca6b4f8945bd0e9d51a"
format_stamp: "Formatted at 2026-06-20 23:14:04 on dist-test-slave-jkbq"
server_key: "e6ad3acdcd52891fb66501af84f1545e"
server_key_iv: "350746c5227e4539537a1835b0bcdb35"
server_key_version: "encryptionkey@0"
I20260620 23:14:04.551716 24545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-1/wal/instance:
uuid: "a8ad0769ea444ca6b4f8945bd0e9d51a"
format_stamp: "Formatted at 2026-06-20 23:14:04 on dist-test-slave-jkbq"
server_key: "e6ad3acdcd52891fb66501af84f1545e"
server_key_iv: "350746c5227e4539537a1835b0bcdb35"
server_key_version: "encryptionkey@0"
I20260620 23:14:04.556805 24545 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:14:04.560246 24560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:04.561553 24545 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260620 23:14:04.561722 24545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-1/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-1/wal
uuid: "a8ad0769ea444ca6b4f8945bd0e9d51a"
format_stamp: "Formatted at 2026-06-20 23:14:04 on dist-test-slave-jkbq"
server_key: "e6ad3acdcd52891fb66501af84f1545e"
server_key_iv: "350746c5227e4539537a1835b0bcdb35"
server_key_version: "encryptionkey@0"
I20260620 23:14:04.561906 24545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:14:04.582399 24545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:14:04.586722 24545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:14:04.587042 24545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:14:04.605639 24545 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.253:33525
I20260620 23:14:04.605680 24611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.253:33525 every 8 connection(s)
I20260620 23:14:04.607098 24545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-1/data/info.pb
I20260620 23:14:04.610805 24612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:14:04.614053 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 24545
I20260620 23:14:04.614245 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-1/wal/instance
I20260620 23:14:04.614675 11007 external_mini_cluster.cc:1428] Setting key cc8710e7e778a3359c4f2b85aedb7e74
I20260620 23:14:04.617421 24612 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.191.254" port: 42913 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:14:04 dist-test-slave-jkbq krb5kdc[24458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997244, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.253@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.191.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.252@KRBTEST.COM" created.
I20260620 23:14:04.639357 24541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:04.620180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53112 (local address 127.10.191.254:42913)
0620 23:14:04.620432 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:04.620441 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:04.621636 (+ 1195us) server_negotiation.cc:407] Connection header received
0620 23:14:04.623542 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:04.623551 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:04.623646 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:04.623774 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:04.625268 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:04.626466 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:04.628048 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:04.628855 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:04.629002 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:04.632980 (+ 3978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:04.632991 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:04.632998 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:04.633045 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:04.636187 (+ 3142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:04.637053 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:04.637058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:04.637060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:04.637181 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:04.637598 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:04.637603 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:04.637605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:04.637889 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:04.638062 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:04.638767 (+ 705us) server_negotiation.cc:299] Negotiation successful
0620 23:14:04.639041 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 23:14:04.640364 24615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:04.620671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:42913 (local address 127.0.0.1:53112)
0620 23:14:04.621460 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:04.621505 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:04.623267 (+ 1762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:04.623992 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:04.624013 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:04.624534 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:14:04.625089 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:04.625106 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:04.626626 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:04.626630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:04.627907 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:04.627918 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:04.629082 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:04.629087 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:04.629291 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:04.629988 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:14:04.630011 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:14:04.632728 (+ 2717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:14:04.636310 (+ 3582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:04.636317 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:04.636330 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:04.636915 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:04.637292 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:04.637297 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:04.637301 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:04.637472 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:04.638016 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:14:04.638024 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:14:04.638402 (+ 378us) client_negotiation.cc:769] Sending connection context
0620 23:14:04.638668 (+ 266us) client_negotiation.cc:240] Negotiation successful
0620 23:14:04.639203 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":347,"threads_started":1}
I20260620 23:14:04.642258 24612 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33525 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:14:04 dist-test-slave-jkbq krb5kdc[24458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997244, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.253@KRBTEST.COM for kudu/127.10.191.253@KRBTEST.COM
I20260620 23:14:04.656876 24615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:04.642940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:33525 (local address 127.0.0.1:37604)
0620 23:14:04.643150 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:04.643171 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:04.643281 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:04.644432 (+ 1151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:04.644436 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:04.644879 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:14:04.645008 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:04.645016 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:04.646397 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:04.646405 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:04.647316 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:04.647328 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:04.648108 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:04.648113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:04.648404 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:04.649020 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:14:04.649037 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:14:04.651289 (+ 2252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:14:04.654397 (+ 3108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:04.654402 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:04.654406 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:04.654757 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:04.655155 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:04.655159 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:04.655163 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:04.655296 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:04.655864 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:14:04.655872 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:14:04.656014 (+ 142us) client_negotiation.cc:769] Sending connection context
0620 23:14:04.656153 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 23:14:04.656464 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":37}
I20260620 23:14:04.656948 24617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:04.643256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37604 (local address 127.10.191.253:33525)
0620 23:14:04.644031 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:04.644037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:04.644061 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:14:04.644105 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:04.644111 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:04.644291 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:04.644441 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:04.645161 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:04.646260 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:04.647444 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:04.647971 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:04.648100 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:04.651480 (+ 3380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:04.651491 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:04.651518 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:04.651550 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:04.654268 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:04.654912 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:04.654920 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:04.654923 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:04.655033 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:04.655399 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:04.655404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:04.655407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:04.655747 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:04.655960 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:04.656299 (+ 339us) server_negotiation.cc:299] Negotiation successful
0620 23:14:04.656580 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":95,"server-negotiator.queue_time_us":609,"thread_start_us":464,"threads_started":1}
I20260620 23:14:04.661166 24612 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.191.252" port: 42595 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:14:04.662680 24615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:04.661970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:42595 (local address 127.0.0.1:56122)
0620 23:14:04.662264 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:04.662431 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:42595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
W20260620 23:14:04.663061 24562 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.191.252:42595: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.252.keytab.
Entry for principal kudu/127.10.191.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.252.keytab.
I20260620 23:14:04.665517 24615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:04.665011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:42595 (local address 127.0.0.1:56126)
0620 23:14:04.665251 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:04.665356 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:42595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260620 23:14:04.665707 24612 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.191.252:42595: Network error: Client connection negotiation failed: client connection to 127.10.191.252:42595: connect: Connection refused (error 111)
I20260620 23:14:04.666575 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.252:42595
--webserver_interface=127.10.191.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.252.keytab
--principal=kudu/127.10.191.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:37381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:42913,127.10.191.253:33525,127.10.191.252:42595
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:14:04.715050 24612 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.191.252" port: 42595 } attempt: 1
I20260620 23:14:04.716329 24615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:04.715878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:42595 (local address 127.0.0.1:56134)
0620 23:14:04.716076 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:04.716180 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:42595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 23:14:04.717263 24615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:04.716928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:42595 (local address 127.0.0.1:56150)
0620 23:14:04.717095 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:04.717159 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:42595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 23:14:04.717446 24612 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.191.252:42595: Network error: Client connection negotiation failed: client connection to 127.10.191.252:42595: connect: Connection refused (error 111)
I20260620 23:14:04.785667 24612 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.191.252" port: 42595 } attempt: 2
I20260620 23:14:04.786952 24615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:04.786341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:42595 (local address 127.0.0.1:56158)
0620 23:14:04.786592 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:04.786709 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:42595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 23:14:04.787995 24615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:04.787613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:42595 (local address 127.0.0.1:56170)
0620 23:14:04.787772 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:04.787860 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:42595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 23:14:04.788295 24612 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.191.252:42595: Network error: Client connection negotiation failed: client connection to 127.10.191.252:42595: connect: Connection refused (error 111)
I20260620 23:14:04.815382 24538 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33525 } attempt: 5
Jun 20 23:14:04 dist-test-slave-jkbq krb5kdc[24458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997244, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for kudu/127.10.191.253@KRBTEST.COM
I20260620 23:14:04.829870 24540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:04.816203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:33525 (local address 127.0.0.1:37614)
0620 23:14:04.816481 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:04.816501 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:04.816613 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:04.816956 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:04.816960 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:04.817329 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:14:04.817534 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:04.817544 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:04.818975 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:04.818979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:04.819813 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:04.819824 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:04.821019 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:04.821024 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:04.821176 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:04.821810 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:14:04.821828 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:14:04.823869 (+ 2041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:14:04.827696 (+ 3827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:04.827701 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:04.827704 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:04.828064 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:04.828443 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:04.828447 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:04.828449 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:04.828577 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:04.829146 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:14:04.829149 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:14:04.829287 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 23:14:04.829419 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 23:14:04.829613 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 23:14:04.830062 24617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:04.816359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37614 (local address 127.10.191.253:33525)
0620 23:14:04.816596 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:04.816604 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:04.816631 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:14:04.816721 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:04.816726 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:04.816808 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:04.816927 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:04.817707 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:04.818782 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:04.819973 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:04.820851 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:04.821010 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:04.824098 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:04.824106 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:04.824112 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:04.824160 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:04.827546 (+ 3386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:04.828181 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:04.828186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:04.828190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:04.828333 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:04.828674 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:04.828678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:04.828681 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:04.829037 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:04.829186 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:04.829541 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 23:14:04.829761 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 23:14:04.831133 24538 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.191.252" port: 42595 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:14:04.832098 24540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:04.831735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:42595 (local address 127.0.0.1:56184)
0620 23:14:04.831897 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:04.831971 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:42595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 23:14:04.832909 24540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:04.832600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:42595 (local address 127.0.0.1:56190)
0620 23:14:04.832724 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:04.832808 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:42595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 23:14:04.833101 24538 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.191.252:42595: Network error: Client connection negotiation failed: client connection to 127.10.191.252:42595: connect: Connection refused (error 111)
I20260620 23:14:04.879541 24612 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.191.252" port: 42595 } attempt: 3
I20260620 23:14:04.880905 24615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:04.880239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:42595 (local address 127.0.0.1:56206)
0620 23:14:04.880522 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:04.880664 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:42595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 23:14:04.882071 24615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:04.881592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:42595 (local address 127.0.0.1:56212)
0620 23:14:04.881841 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:04.881930 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:42595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
W20260620 23:14:04.882309 24612 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.191.252:42595: Network error: Client connection negotiation failed: client connection to 127.10.191.252:42595: connect: Connection refused (error 111)
I20260620 23:14:04.884317 24538 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.191.252" port: 42595 } attempt: 1
I20260620 23:14:04.885417 24540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:04.884877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:42595 (local address 127.0.0.1:56214)
0620 23:14:04.885126 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:04.885231 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:42595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 23:14:04.886453 24540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:04.886047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:42595 (local address 127.0.0.1:56218)
0620 23:14:04.886219 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:04.886321 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:42595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 23:14:04.886767 24538 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.191.252:42595: Network error: Client connection negotiation failed: client connection to 127.10.191.252:42595: connect: Connection refused (error 111)
I20260620 23:14:04.954975 24538 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.191.252" port: 42595 } attempt: 2
I20260620 23:14:04.956281 24540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:04.955645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:42595 (local address 127.0.0.1:56222)
0620 23:14:04.955930 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:04.956061 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:42595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 23:14:04.957458 24540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:04.957038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:42595 (local address 127.0.0.1:56234)
0620 23:14:04.957195 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:04.957297 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:42595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 23:14:04.957679 24538 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.191.252:42595: Network error: Client connection negotiation failed: client connection to 127.10.191.252:42595: connect: Connection refused (error 111)
W20260620 23:14:04.993510 24619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:14:04.993781 24619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:14:04.993870 24619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:14:05.000931 24619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:14:05.001032 24619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:14:05.001078 24619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:14:05.001135 24619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:14:05.001178 24619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:14:05.010757 24619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:37381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:42913,127.10.191.253:33525,127.10.191.252:42595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.252.keytab
--principal=kudu/127.10.191.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.252:42595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.191.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:14:05.011972 24619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:14:05.013163 24619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:14:05.024295 24624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:05.024326 24625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:05.025064 24627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:14:05.025516 24612 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.191.252" port: 42595 } attempt: 4
I20260620 23:14:05.026633 24619 server_base.cc:1034] running on GCE node
I20260620 23:14:05.026767 24615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:05.026154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:42595 (local address 127.0.0.1:56240)
0620 23:14:05.026425 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:05.026563 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:42595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 23:14:05.027541 24619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:14:05.027696 24615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:05.027354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:42595 (local address 127.0.0.1:56256)
0620 23:14:05.027501 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:05.027575 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:42595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 23:14:05.027916 24612 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.191.252:42595: Network error: Client connection negotiation failed: client connection to 127.10.191.252:42595: connect: Connection refused (error 111)
I20260620 23:14:05.028995 24619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:14:05.030215 24619 hybrid_clock.cc:648] HybridClock initialized: now 1781997245030172 us; error 51 us; skew 500 ppm
I20260620 23:14:05.030556 24619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:14:05 dist-test-slave-jkbq krb5kdc[24458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997245, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:14:05.033536 24619 init.cc:377] Logged in from keytab as kudu/127.10.191.252@KRBTEST.COM (short username kudu)
I20260620 23:14:05.034960 24619 webserver.cc:460] Webserver started at http://127.10.191.252:33995/ using document root <none> and password file <none>
I20260620 23:14:05.035634 24619 fs_manager.cc:359] Metadata directory not provided
I20260620 23:14:05.035769 24619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:14:05.036060 24619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:14:05.039002 24619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-2/data/instance:
uuid: "04978d32ef074ba5b3f905538105d712"
format_stamp: "Formatted at 2026-06-20 23:14:05 on dist-test-slave-jkbq"
server_key: "d14eb362d3e3a4711b7ea1b6f64844fc"
server_key_iv: "9dc8ac7735f6543de7cb1c03f2aee03b"
server_key_version: "encryptionkey@0"
I20260620 23:14:05.039776 24619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-2/wal/instance:
uuid: "04978d32ef074ba5b3f905538105d712"
format_stamp: "Formatted at 2026-06-20 23:14:05 on dist-test-slave-jkbq"
server_key: "d14eb362d3e3a4711b7ea1b6f64844fc"
server_key_iv: "9dc8ac7735f6543de7cb1c03f2aee03b"
server_key_version: "encryptionkey@0"
I20260620 23:14:05.044642 24619 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:14:05.047957 24634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:05.049237 24619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:14:05.049464 24619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-2/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-2/wal
uuid: "04978d32ef074ba5b3f905538105d712"
format_stamp: "Formatted at 2026-06-20 23:14:05 on dist-test-slave-jkbq"
server_key: "d14eb362d3e3a4711b7ea1b6f64844fc"
server_key_iv: "9dc8ac7735f6543de7cb1c03f2aee03b"
server_key_version: "encryptionkey@0"
I20260620 23:14:05.049660 24619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:14:05.063874 24538 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.191.252" port: 42595 } attempt: 3
I20260620 23:14:05.064810 24540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:05.064362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:42595 (local address 127.0.0.1:56270)
0620 23:14:05.064579 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:05.064658 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:42595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 23:14:05.065783 24540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:05.065409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:42595 (local address 127.0.0.1:56274)
0620 23:14:05.065615 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:05.065677 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:42595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260620 23:14:05.065985 24538 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.191.252:42595: Network error: Client connection negotiation failed: client connection to 127.10.191.252:42595: connect: Connection refused (error 111)
I20260620 23:14:05.074990 24619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:14:05.078977 24619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:14:05.079237 24619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:14:05.097328 24619 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.252:42595
I20260620 23:14:05.097359 24685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.252:42595 every 8 connection(s)
I20260620 23:14:05.098649 24619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-2/data/info.pb
I20260620 23:14:05.102227 24686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:14:05.108718 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 24619
I20260620 23:14:05.108237 24686 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.191.254" port: 42913 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:14:05.108908 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781997042893763-11007-0/minicluster-data/master-2/wal/instance
I20260620 23:14:05.109345 11007 external_mini_cluster.cc:1428] Setting key fb649948f9c98e5b31548b9cdc626ed6
Jun 20 23:14:05 dist-test-slave-jkbq krb5kdc[24458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997245, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.252@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
Jun 20 23:14:05 dist-test-slave-jkbq krb5kdc[24458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.252@KRBTEST.COM
I20260620 23:14:05.129920 24541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:05.111173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53114 (local address 127.10.191.254:42913)
0620 23:14:05.111412 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:05.111421 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:05.112120 (+ 699us) server_negotiation.cc:407] Connection header received
0620 23:14:05.114590 (+ 2470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:05.114599 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:05.114672 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:05.114778 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:05.116155 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:05.117281 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:05.119424 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:05.120100 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:05.120211 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:05.123608 (+ 3397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:05.123613 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:05.123617 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:05.123647 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:05.126614 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:05.127540 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:05.127544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:05.127546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:05.127679 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:05.128137 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:05.128143 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:05.128148 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:05.128508 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:05.128675 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:05.129449 (+ 774us) server_negotiation.cc:299] Negotiation successful
0620 23:14:05.129608 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 23:14:05.131140 24689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:05.111047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:42913 (local address 127.0.0.1:53114)
0620 23:14:05.111966 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:05.112023 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:05.114336 (+ 2313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:05.114833 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:05.114843 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:05.115444 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:14:05.115984 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:05.116011 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:05.117450 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:05.117455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:05.119279 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:05.119293 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:05.120508 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:05.120513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:05.120617 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:05.121294 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:14:05.121314 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:14:05.123424 (+ 2110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:14:05.126739 (+ 3315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:05.126751 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:05.126764 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:05.127423 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:05.127790 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:05.127795 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:05.127798 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:05.128020 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:05.128638 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:14:05.128644 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:14:05.129052 (+ 408us) client_negotiation.cc:769] Sending connection context
0620 23:14:05.129339 (+ 287us) client_negotiation.cc:240] Negotiation successful
0620 23:14:05.129800 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":6,"thread_start_us":395,"threads_started":1}
I20260620 23:14:05.131170 24690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:05.111047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56286 (local address 127.10.191.252:42595)
0620 23:14:05.111998 (+ 951us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:05.112012 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:05.112056 (+ 44us) server_negotiation.cc:407] Connection header received
0620 23:14:05.114328 (+ 2272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:05.114353 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:05.114623 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:05.116212 (+ 1589us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:05.116384 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:05.117835 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:05.119224 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:05.120054 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:05.120230 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:05.124913 (+ 4683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:05.124921 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:05.124932 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:05.124958 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:05.127912 (+ 2954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:05.128769 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:05.128775 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:05.128778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:05.128922 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:05.129255 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:05.129260 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:05.129263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:05.129670 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:05.129938 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:05.130312 (+ 374us) server_negotiation.cc:299] Negotiation successful
0620 23:14:05.130580 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":704,"thread_start_us":494,"threads_started":1}
I20260620 23:14:05.132658 24686 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33525 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:14:05 dist-test-slave-jkbq krb5kdc[24458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997245, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.252@KRBTEST.COM for kudu/127.10.191.253@KRBTEST.COM
Jun 20 23:14:05 dist-test-slave-jkbq krb5kdc[24458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:14:05.145565 24689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:05.133243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:33525 (local address 127.0.0.1:37624)
0620 23:14:05.133408 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:05.133425 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:05.133519 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:05.133912 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:05.133915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:05.134209 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:14:05.134372 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:05.134384 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:05.135731 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:05.135735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:05.136616 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:05.136629 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:05.137329 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:05.137335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:05.137568 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:05.138092 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:14:05.138108 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:14:05.140278 (+ 2170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:14:05.143272 (+ 2994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:05.143277 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:05.143281 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:05.143679 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:05.144052 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:05.144057 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:05.144061 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:05.144195 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:05.144672 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:14:05.144677 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:14:05.144817 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 23:14:05.144959 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 23:14:05.145215 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 23:14:05.145612 24617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:05.133465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37624 (local address 127.10.191.253:33525)
0620 23:14:05.133666 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:05.133670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:05.133686 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:14:05.133733 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:05.133737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:05.133791 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:05.133865 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:05.134526 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:05.135600 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:05.136752 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:05.137173 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:05.137294 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:05.140454 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:05.140458 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:05.140462 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:05.140491 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:05.143153 (+ 2662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:05.143802 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:05.143807 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:05.143811 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:05.143944 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:05.144299 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:05.144305 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:05.144308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:05.144556 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:05.144720 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:05.145058 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 23:14:05.145253 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 23:14:05.146651 24686 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.191.252" port: 42595 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:14:05.148983 24541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:05.134531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53130 (local address 127.10.191.254:42913)
0620 23:14:05.134706 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:05.134711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:05.135145 (+ 434us) server_negotiation.cc:407] Connection header received
0620 23:14:05.135272 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:05.135275 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:05.135325 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:05.135397 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:05.136614 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:05.137426 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:05.138775 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:05.139374 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:05.139466 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:05.143462 (+ 3996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:05.143467 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:05.143470 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:05.143499 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:05.146163 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:05.146959 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:05.146965 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:05.146968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:05.147128 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:05.147556 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:05.147561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:05.147564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:05.147882 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:05.148161 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:05.148473 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 23:14:05.148686 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 23:14:05 dist-test-slave-jkbq krb5kdc[24458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997245, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.252@KRBTEST.COM for kudu/127.10.191.252@KRBTEST.COM
Jun 20 23:14:05 dist-test-slave-jkbq krb5kdc[24458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.253@KRBTEST.COM
I20260620 23:14:05.160950 24689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:05.147661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:42595 (local address 127.0.0.1:56296)
0620 23:14:05.147894 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:05.147907 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:05.148006 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:05.148264 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:05.148268 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:05.148613 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:14:05.148731 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:05.148738 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:05.150058 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:05.150061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:05.151120 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:05.151132 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:05.152069 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:05.152073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:05.152244 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:05.152844 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:14:05.152867 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:14:05.155127 (+ 2260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:14:05.158623 (+ 3496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:05.158627 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:05.158629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:05.159084 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:05.159416 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:05.159422 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:05.159425 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:05.159593 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:05.160069 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:14:05.160075 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:14:05.160252 (+ 177us) client_negotiation.cc:769] Sending connection context
0620 23:14:05.160415 (+ 163us) client_negotiation.cc:240] Negotiation successful
0620 23:14:05.160692 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":2}
I20260620 23:14:05.161324 24690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:05.147654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56296 (local address 127.10.191.252:42595)
0620 23:14:05.147812 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:05.147818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:05.147979 (+ 161us) server_negotiation.cc:407] Connection header received
0620 23:14:05.148106 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:05.148111 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:05.148163 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:05.148250 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:05.148990 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:05.149946 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:05.151232 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:05.151932 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:05.152079 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:05.155287 (+ 3208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:05.155292 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:05.155295 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:05.155321 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:05.158488 (+ 3167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:05.159189 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:05.159193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:05.159195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:05.159309 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:05.159702 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:05.159705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:05.159707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:05.159951 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:05.160058 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:05.160521 (+ 463us) server_negotiation.cc:299] Negotiation successful
0620 23:14:05.160856 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":130,"server-negotiator.queue_time_us":52}
I20260620 23:14:05.166154 24617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:05.150524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37630 (local address 127.10.191.253:33525)
0620 23:14:05.150671 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:05.150678 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:05.150725 (+ 47us) server_negotiation.cc:407] Connection header received
0620 23:14:05.150911 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:05.150920 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:05.150969 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:05.151061 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:05.152549 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:05.153650 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:05.155091 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:05.155555 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:05.155644 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:05.160604 (+ 4960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:05.160611 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:05.160616 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:05.160653 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:05.163544 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:05.164315 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:05.164319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:05.164321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:05.164478 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:05.164863 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:05.164866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:05.164868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:05.165114 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:05.165369 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:05.165651 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 23:14:05.165866 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 23:14:05.167085 24686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04978d32ef074ba5b3f905538105d712: Bootstrap starting.
I20260620 23:14:05.170137 24686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04978d32ef074ba5b3f905538105d712: Neither blocks nor log segments found. Creating new log.
I20260620 23:14:05.170436 11007 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:14:05.171247 24686 log.cc:824] T 00000000000000000000000000000000 P 04978d32ef074ba5b3f905538105d712: Log is configured to *not* fsync() on all Append() calls
I20260620 23:14:05.173985 24686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04978d32ef074ba5b3f905538105d712: No bootstrap required, opened a new log
I20260620 23:14:05.185482 24686 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04978d32ef074ba5b3f905538105d712 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c46bab565ee4f9aa2bc4ff19aab2f19" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 42913 } } peers { permanent_uuid: "a8ad0769ea444ca6b4f8945bd0e9d51a" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33525 } } peers { permanent_uuid: "04978d32ef074ba5b3f905538105d712" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 42595 } }
I20260620 23:14:05.186131 24686 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04978d32ef074ba5b3f905538105d712 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:14:05.186322 24686 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04978d32ef074ba5b3f905538105d712 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04978d32ef074ba5b3f905538105d712, State: Initialized, Role: FOLLOWER
I20260620 23:14:05.187275 24686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04978d32ef074ba5b3f905538105d712 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c46bab565ee4f9aa2bc4ff19aab2f19" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 42913 } } peers { permanent_uuid: "a8ad0769ea444ca6b4f8945bd0e9d51a" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33525 } } peers { permanent_uuid: "04978d32ef074ba5b3f905538105d712" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 42595 } }
I20260620 23:14:05.197781 24541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:05.175396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53134 (local address 127.10.191.254:42913)
0620 23:14:05.175580 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:05.175587 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:05.176575 (+ 988us) server_negotiation.cc:407] Connection header received
0620 23:14:05.176823 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:05.176829 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:05.176907 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:05.176978 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:05.179428 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:05.180536 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:05.182682 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:05.183416 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:05.183518 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:05.188392 (+ 4874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:05.188402 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:05.188410 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:05.188478 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:05.193237 (+ 4759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:05.194555 (+ 1318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:05.194564 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:05.194570 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:05.194813 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:05.195425 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:05.195433 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:05.195437 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:05.195899 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:05.196184 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:05.196913 (+ 729us) server_negotiation.cc:299] Negotiation successful
0620 23:14:05.197242 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 23:14:05.199717 24699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04978d32ef074ba5b3f905538105d712 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c46bab565ee4f9aa2bc4ff19aab2f19" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 42913 } } peers { permanent_uuid: "a8ad0769ea444ca6b4f8945bd0e9d51a" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33525 } } peers { permanent_uuid: "04978d32ef074ba5b3f905538105d712" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 42595 } } }
I20260620 23:14:05.200258 24699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04978d32ef074ba5b3f905538105d712 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:14:05.201237 24686 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04978d32ef074ba5b3f905538105d712 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:14:05.201403 24690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:05.175795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56306 (local address 127.10.191.252:42595)
0620 23:14:05.175987 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:05.175994 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:05.183375 (+ 7381us) server_negotiation.cc:407] Connection header received
0620 23:14:05.183515 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:05.183523 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:05.183630 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:05.183759 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:05.186371 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:05.187558 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:05.189674 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:05.190429 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:05.190542 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:05.194215 (+ 3673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:05.194223 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:05.194231 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:05.194272 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:05.197968 (+ 3696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:05.198823 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:05.198829 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:05.198833 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:05.199012 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:05.199456 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:05.199466 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:05.199469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:05.199915 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:05.200151 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:05.200767 (+ 616us) server_negotiation.cc:299] Negotiation successful
0620 23:14:05.200969 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 23:14:05.204732 24617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:05.176982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37646 (local address 127.10.191.253:33525)
0620 23:14:05.177198 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:05.177206 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:05.185803 (+ 8597us) server_negotiation.cc:407] Connection header received
0620 23:14:05.185948 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:05.185953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:05.186041 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:05.186144 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:05.188115 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:05.189316 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:05.191317 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:05.192020 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:05.192145 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:05.195793 (+ 3648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:05.195799 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:05.195805 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:05.195845 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:05.201181 (+ 5336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:05.201998 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:05.202003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:05.202006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:05.202141 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:05.202525 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:05.202535 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:05.202555 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:05.202995 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:05.204170 (+ 1175us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:05.204266 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 23:14:05.204429 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260620 23:14:05.211166 24710 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 04978d32ef074ba5b3f905538105d712: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:14:05.211302 24710 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:14:05.208427 24619 master_runner.cc:186] Error getting master registration for 127.10.191.254:42913: OK, instance_id { permanent_uuid: "6c46bab565ee4f9aa2bc4ff19aab2f19" instance_seqno: 1781997244107568 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 23:14:05.216856 24619 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 23:14:05.243229 24538 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.191.252" port: 42595 } attempt: 4
Jun 20 23:14:05 dist-test-slave-jkbq krb5kdc[24458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997244, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for kudu/127.10.191.252@KRBTEST.COM
I20260620 23:14:05.257299 24690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:05.243962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56318 (local address 127.10.191.252:42595)
0620 23:14:05.244136 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:05.244142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:05.244206 (+ 64us) server_negotiation.cc:407] Connection header received
0620 23:14:05.244412 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:05.244417 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:05.244503 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:05.244608 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:05.245383 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:05.246227 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:05.247914 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:05.248371 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:05.248502 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:05.251532 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:05.251537 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:05.251540 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:05.251571 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:05.254681 (+ 3110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:05.255342 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:05.255350 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:05.255353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:05.255491 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:05.255886 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:05.255890 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:05.255903 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:05.256200 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:05.256350 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:05.256722 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 23:14:05.256953 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:14:05.257292 24540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:05.243908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:42595 (local address 127.0.0.1:56318)
0620 23:14:05.244107 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:05.244130 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:05.244263 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:05.244635 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:05.244640 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:05.245080 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:14:05.245240 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:05.245249 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:05.246386 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:05.246394 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:05.247780 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:05.247793 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:05.248509 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:05.248515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:05.248701 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:05.249307 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:14:05.249326 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:14:05.251361 (+ 2035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:14:05.254816 (+ 3455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:05.254823 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:05.254827 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:05.255225 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:05.255606 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:05.255612 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:05.255616 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:05.255770 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:05.256318 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:14:05.256323 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:14:05.256486 (+ 163us) client_negotiation.cc:769] Sending connection context
0620 23:14:05.256619 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 23:14:05.256896 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 23:14:05.262681 24538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c46bab565ee4f9aa2bc4ff19aab2f19: Bootstrap starting.
I20260620 23:14:05.266058 24538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c46bab565ee4f9aa2bc4ff19aab2f19: Neither blocks nor log segments found. Creating new log.
I20260620 23:14:05.267251 24538 log.cc:824] T 00000000000000000000000000000000 P 6c46bab565ee4f9aa2bc4ff19aab2f19: Log is configured to *not* fsync() on all Append() calls
I20260620 23:14:05.271080 24538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c46bab565ee4f9aa2bc4ff19aab2f19: No bootstrap required, opened a new log
I20260620 23:14:05.278198 24538 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c46bab565ee4f9aa2bc4ff19aab2f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c46bab565ee4f9aa2bc4ff19aab2f19" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 42913 } } peers { permanent_uuid: "a8ad0769ea444ca6b4f8945bd0e9d51a" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33525 } } peers { permanent_uuid: "04978d32ef074ba5b3f905538105d712" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 42595 } }
I20260620 23:14:05.278508 24538 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c46bab565ee4f9aa2bc4ff19aab2f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:14:05.278661 24538 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c46bab565ee4f9aa2bc4ff19aab2f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c46bab565ee4f9aa2bc4ff19aab2f19, State: Initialized, Role: FOLLOWER
I20260620 23:14:05.279326 24538 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c46bab565ee4f9aa2bc4ff19aab2f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c46bab565ee4f9aa2bc4ff19aab2f19" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 42913 } } peers { permanent_uuid: "a8ad0769ea444ca6b4f8945bd0e9d51a" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33525 } } peers { permanent_uuid: "04978d32ef074ba5b3f905538105d712" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 42595 } }
I20260620 23:14:05.280306 24713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c46bab565ee4f9aa2bc4ff19aab2f19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c46bab565ee4f9aa2bc4ff19aab2f19" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 42913 } } peers { permanent_uuid: "a8ad0769ea444ca6b4f8945bd0e9d51a" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33525 } } peers { permanent_uuid: "04978d32ef074ba5b3f905538105d712" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 42595 } } }
I20260620 23:14:05.280694 24713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c46bab565ee4f9aa2bc4ff19aab2f19 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:14:05.281606 24538 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c46bab565ee4f9aa2bc4ff19aab2f19 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 23:14:05.293702 24724 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6c46bab565ee4f9aa2bc4ff19aab2f19: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:14:05.293932 24724 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:14:05.292268 24471 master_runner.cc:186] Error getting master registration for 127.10.191.253:33525: OK, instance_id { permanent_uuid: "a8ad0769ea444ca6b4f8945bd0e9d51a" instance_seqno: 1781997244600164 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 23:14:05.302628 24471 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 23:14:05.327112 24612 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.191.252" port: 42595 } attempt: 5
W20260620 23:14:05.331277 24640 tablet.cc:2369] T 00000000000000000000000000000000 P 04978d32ef074ba5b3f905538105d712: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 23:14:05 dist-test-slave-jkbq krb5kdc[24458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997244, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.253@KRBTEST.COM for kudu/127.10.191.252@KRBTEST.COM
I20260620 23:14:05.341944 24690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:05.327812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56330 (local address 127.10.191.252:42595)
0620 23:14:05.328000 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:05.328007 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:05.328072 (+ 65us) server_negotiation.cc:407] Connection header received
0620 23:14:05.328275 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:05.328281 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:05.328351 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:05.328449 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:05.329239 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:05.329997 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:05.331726 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:05.332380 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:05.332523 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:05.336566 (+ 4043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:05.336570 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:05.336574 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:05.336603 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:05.339399 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:05.340021 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:05.340026 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:05.340030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:05.340154 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:05.340542 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:05.340548 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:05.340551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:05.340832 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:05.340977 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:05.341338 (+ 361us) server_negotiation.cc:299] Negotiation successful
0620 23:14:05.341576 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:14:05.341934 24615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:05.327753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:42595 (local address 127.0.0.1:56330)
0620 23:14:05.327998 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:05.328017 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:05.328147 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:05.328480 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:05.328484 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:05.328918 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:14:05.329081 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:05.329089 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:05.330142 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:05.330148 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:05.331480 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:05.331494 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:05.333469 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:05.333478 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:05.333705 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:05.334317 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:14:05.334334 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:14:05.336369 (+ 2035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:14:05.339533 (+ 3164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:05.339539 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:05.339543 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:05.339901 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:05.340258 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:05.340263 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:05.340266 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:05.340427 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:05.340955 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:14:05.340959 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:14:05.341104 (+ 145us) client_negotiation.cc:769] Sending connection context
0620 23:14:05.341240 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 23:14:05.341533 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260620 23:14:05.346609 24612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8ad0769ea444ca6b4f8945bd0e9d51a: Bootstrap starting.
W20260620 23:14:05.347297 24492 tablet.cc:2369] T 00000000000000000000000000000000 P 6c46bab565ee4f9aa2bc4ff19aab2f19: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:14:05.350419 24612 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8ad0769ea444ca6b4f8945bd0e9d51a: Neither blocks nor log segments found. Creating new log.
I20260620 23:14:05.351856 24612 log.cc:824] T 00000000000000000000000000000000 P a8ad0769ea444ca6b4f8945bd0e9d51a: Log is configured to *not* fsync() on all Append() calls
I20260620 23:14:05.355751 24612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8ad0769ea444ca6b4f8945bd0e9d51a: No bootstrap required, opened a new log
I20260620 23:14:05.362807 24612 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8ad0769ea444ca6b4f8945bd0e9d51a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c46bab565ee4f9aa2bc4ff19aab2f19" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 42913 } } peers { permanent_uuid: "a8ad0769ea444ca6b4f8945bd0e9d51a" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33525 } } peers { permanent_uuid: "04978d32ef074ba5b3f905538105d712" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 42595 } }
I20260620 23:14:05.363221 24612 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8ad0769ea444ca6b4f8945bd0e9d51a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:14:05.363338 24612 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8ad0769ea444ca6b4f8945bd0e9d51a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8ad0769ea444ca6b4f8945bd0e9d51a, State: Initialized, Role: FOLLOWER
I20260620 23:14:05.364032 24612 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8ad0769ea444ca6b4f8945bd0e9d51a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c46bab565ee4f9aa2bc4ff19aab2f19" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 42913 } } peers { permanent_uuid: "a8ad0769ea444ca6b4f8945bd0e9d51a" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33525 } } peers { permanent_uuid: "04978d32ef074ba5b3f905538105d712" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 42595 } }
I20260620 23:14:05.365074 24727 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8ad0769ea444ca6b4f8945bd0e9d51a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c46bab565ee4f9aa2bc4ff19aab2f19" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 42913 } } peers { permanent_uuid: "a8ad0769ea444ca6b4f8945bd0e9d51a" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33525 } } peers { permanent_uuid: "04978d32ef074ba5b3f905538105d712" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 42595 } } }
I20260620 23:14:05.365686 24727 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8ad0769ea444ca6b4f8945bd0e9d51a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:14:05.366408 24612 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8ad0769ea444ca6b4f8945bd0e9d51a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:14:05.390995 24545 master_runner.cc:263] No leader master found from master a8ad0769ea444ca6b4f8945bd0e9d51a
I20260620 23:14:05.391201 24545 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 23:14:05.397604 24738 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a8ad0769ea444ca6b4f8945bd0e9d51a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:14:05.397766 24738 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:14:05.415845 24713 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c46bab565ee4f9aa2bc4ff19aab2f19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:14:05.416181 24713 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c46bab565ee4f9aa2bc4ff19aab2f19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c46bab565ee4f9aa2bc4ff19aab2f19" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 42913 } } peers { permanent_uuid: "a8ad0769ea444ca6b4f8945bd0e9d51a" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33525 } } peers { permanent_uuid: "04978d32ef074ba5b3f905538105d712" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 42595 } }
I20260620 23:14:05.417817 24713 leader_election.cc:288] T 00000000000000000000000000000000 P 6c46bab565ee4f9aa2bc4ff19aab2f19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8ad0769ea444ca6b4f8945bd0e9d51a (127.10.191.253:33525), 04978d32ef074ba5b3f905538105d712 (127.10.191.252:42595)
I20260620 23:14:05.419023 24587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6c46bab565ee4f9aa2bc4ff19aab2f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8ad0769ea444ca6b4f8945bd0e9d51a" is_pre_election: true
I20260620 23:14:05.419600 24587 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a8ad0769ea444ca6b4f8945bd0e9d51a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c46bab565ee4f9aa2bc4ff19aab2f19 in term 0.
I20260620 23:14:05.421103 24489 leader_election.cc:302] T 00000000000000000000000000000000 P 6c46bab565ee4f9aa2bc4ff19aab2f19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c46bab565ee4f9aa2bc4ff19aab2f19, a8ad0769ea444ca6b4f8945bd0e9d51a; no voters:
I20260620 23:14:05.421865 24661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6c46bab565ee4f9aa2bc4ff19aab2f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04978d32ef074ba5b3f905538105d712" is_pre_election: true
I20260620 23:14:05.422441 24661 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 04978d32ef074ba5b3f905538105d712 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c46bab565ee4f9aa2bc4ff19aab2f19 in term 0.
I20260620 23:14:05.426230 24713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c46bab565ee4f9aa2bc4ff19aab2f19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:14:05.426404 24713 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c46bab565ee4f9aa2bc4ff19aab2f19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:14:05.426497 24713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c46bab565ee4f9aa2bc4ff19aab2f19 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:14:05.429949 24713 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c46bab565ee4f9aa2bc4ff19aab2f19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c46bab565ee4f9aa2bc4ff19aab2f19" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 42913 } } peers { permanent_uuid: "a8ad0769ea444ca6b4f8945bd0e9d51a" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33525 } } peers { permanent_uuid: "04978d32ef074ba5b3f905538105d712" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 42595 } }
I20260620 23:14:05.431082 24587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6c46bab565ee4f9aa2bc4ff19aab2f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8ad0769ea444ca6b4f8945bd0e9d51a"
I20260620 23:14:05.431291 24587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8ad0769ea444ca6b4f8945bd0e9d51a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:14:05.434645 24587 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a8ad0769ea444ca6b4f8945bd0e9d51a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c46bab565ee4f9aa2bc4ff19aab2f19 in term 1.
I20260620 23:14:05.435175 24489 leader_election.cc:302] T 00000000000000000000000000000000 P 6c46bab565ee4f9aa2bc4ff19aab2f19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c46bab565ee4f9aa2bc4ff19aab2f19, a8ad0769ea444ca6b4f8945bd0e9d51a; no voters:
I20260620 23:14:05.437808 24661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6c46bab565ee4f9aa2bc4ff19aab2f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04978d32ef074ba5b3f905538105d712"
I20260620 23:14:05.438030 24661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04978d32ef074ba5b3f905538105d712 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:14:05.441439 24661 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 04978d32ef074ba5b3f905538105d712 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c46bab565ee4f9aa2bc4ff19aab2f19 in term 1.
I20260620 23:14:05.442154 24713 leader_election.cc:288] T 00000000000000000000000000000000 P 6c46bab565ee4f9aa2bc4ff19aab2f19 [CANDIDATE]: Term 1 election: Requested vote from peers a8ad0769ea444ca6b4f8945bd0e9d51a (127.10.191.253:33525), 04978d32ef074ba5b3f905538105d712 (127.10.191.252:42595)
I20260620 23:14:05.442595 24713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c46bab565ee4f9aa2bc4ff19aab2f19 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:14:05.442979 24713 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c46bab565ee4f9aa2bc4ff19aab2f19 [term 1 LEADER]: Becoming Leader. State: Replica: 6c46bab565ee4f9aa2bc4ff19aab2f19, State: Running, Role: LEADER
I20260620 23:14:05.447962 24713 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c46bab565ee4f9aa2bc4ff19aab2f19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c46bab565ee4f9aa2bc4ff19aab2f19" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 42913 } } peers { permanent_uuid: "a8ad0769ea444ca6b4f8945bd0e9d51a" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33525 } } peers { permanent_uuid: "04978d32ef074ba5b3f905538105d712" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 42595 } }
I20260620 23:14:05.463390 24740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c46bab565ee4f9aa2bc4ff19aab2f19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c46bab565ee4f9aa2bc4ff19aab2f19. Latest consensus state: current_term: 1 leader_uuid: "6c46bab565ee4f9aa2bc4ff19aab2f19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c46bab565ee4f9aa2bc4ff19aab2f19" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 42913 } } peers { permanent_uuid: "a8ad0769ea444ca6b4f8945bd0e9d51a" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33525 } } peers { permanent_uuid: "04978d32ef074ba5b3f905538105d712" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 42595 } } }
I20260620 23:14:05.464310 24740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c46bab565ee4f9aa2bc4ff19aab2f19 [sys.catalog]: This master's current role is: LEADER
I20260620 23:14:05.471449 24742 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:14:05.474071 24742 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:14:05.493332 24661 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 04978d32ef074ba5b3f905538105d712 [term 1 FOLLOWER]: Refusing update from remote peer 6c46bab565ee4f9aa2bc4ff19aab2f19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:14:05.494385 24740 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6c46bab565ee4f9aa2bc4ff19aab2f19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04978d32ef074ba5b3f905538105d712" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 42595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:14:05.502205 24587 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a8ad0769ea444ca6b4f8945bd0e9d51a [term 1 FOLLOWER]: Refusing update from remote peer 6c46bab565ee4f9aa2bc4ff19aab2f19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:14:05.503224 24740 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6c46bab565ee4f9aa2bc4ff19aab2f19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8ad0769ea444ca6b4f8945bd0e9d51a" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:14:05.541672 24699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04978d32ef074ba5b3f905538105d712 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c46bab565ee4f9aa2bc4ff19aab2f19. Latest consensus state: current_term: 1 leader_uuid: "6c46bab565ee4f9aa2bc4ff19aab2f19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c46bab565ee4f9aa2bc4ff19aab2f19" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 42913 } } peers { permanent_uuid: "a8ad0769ea444ca6b4f8945bd0e9d51a" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33525 } } peers { permanent_uuid: "04978d32ef074ba5b3f905538105d712" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 42595 } } }
I20260620 23:14:05.541980 24699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04978d32ef074ba5b3f905538105d712 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:14:05.545274 24743 mvcc.cc:205] Tried to move back new op lower bound from 7299060717528911872 to 7299060717364649984. Current Snapshot: MvccSnapshot[applied={T|T < 7299060717528911872}]
I20260620 23:14:05.551520 24727 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8ad0769ea444ca6b4f8945bd0e9d51a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c46bab565ee4f9aa2bc4ff19aab2f19. Latest consensus state: current_term: 1 leader_uuid: "6c46bab565ee4f9aa2bc4ff19aab2f19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c46bab565ee4f9aa2bc4ff19aab2f19" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 42913 } } peers { permanent_uuid: "a8ad0769ea444ca6b4f8945bd0e9d51a" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33525 } } peers { permanent_uuid: "04978d32ef074ba5b3f905538105d712" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 42595 } } }
I20260620 23:14:05.551854 24727 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8ad0769ea444ca6b4f8945bd0e9d51a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:14:05.555359 24744 mvcc.cc:205] Tried to move back new op lower bound from 7299060717528911872 to 7299060717364649984. Current Snapshot: MvccSnapshot[applied={T|T < 7299060717528911872}]
I20260620 23:14:05.559758 24713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c46bab565ee4f9aa2bc4ff19aab2f19 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c46bab565ee4f9aa2bc4ff19aab2f19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c46bab565ee4f9aa2bc4ff19aab2f19" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 42913 } } peers { permanent_uuid: "a8ad0769ea444ca6b4f8945bd0e9d51a" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33525 } } peers { permanent_uuid: "04978d32ef074ba5b3f905538105d712" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 42595 } } }
I20260620 23:14:05.560077 24713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c46bab565ee4f9aa2bc4ff19aab2f19 [sys.catalog]: This master's current role is: LEADER
I20260620 23:14:05.560456 24713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c46bab565ee4f9aa2bc4ff19aab2f19 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c46bab565ee4f9aa2bc4ff19aab2f19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c46bab565ee4f9aa2bc4ff19aab2f19" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 42913 } } peers { permanent_uuid: "a8ad0769ea444ca6b4f8945bd0e9d51a" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33525 } } peers { permanent_uuid: "04978d32ef074ba5b3f905538105d712" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 42595 } } }
I20260620 23:14:05.560701 24713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c46bab565ee4f9aa2bc4ff19aab2f19 [sys.catalog]: This master's current role is: LEADER
I20260620 23:14:05.561247 24745 mvcc.cc:205] Tried to move back new op lower bound from 7299060717528911872 to 7299060717364649984. Current Snapshot: MvccSnapshot[applied={T|T < 7299060717528911872}]
I20260620 23:14:05.561482 24699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04978d32ef074ba5b3f905538105d712 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c46bab565ee4f9aa2bc4ff19aab2f19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c46bab565ee4f9aa2bc4ff19aab2f19" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 42913 } } peers { permanent_uuid: "a8ad0769ea444ca6b4f8945bd0e9d51a" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33525 } } peers { permanent_uuid: "04978d32ef074ba5b3f905538105d712" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 42595 } } }
I20260620 23:14:05.561754 24699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04978d32ef074ba5b3f905538105d712 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:14:05.561816 24742 catalog_manager.cc:1347] Generated new cluster ID: 7fb168994d504fdea131ecb96d89a869
I20260620 23:14:05.561904 24742 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:14:05.574726 24727 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8ad0769ea444ca6b4f8945bd0e9d51a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c46bab565ee4f9aa2bc4ff19aab2f19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c46bab565ee4f9aa2bc4ff19aab2f19" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 42913 } } peers { permanent_uuid: "a8ad0769ea444ca6b4f8945bd0e9d51a" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 33525 } } peers { permanent_uuid: "04978d32ef074ba5b3f905538105d712" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 42595 } } }
I20260620 23:14:05.575031 24727 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8ad0769ea444ca6b4f8945bd0e9d51a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:14:05.616891 24742 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:14:05.618702 24742 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:14:05.644455 24742 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c46bab565ee4f9aa2bc4ff19aab2f19: Generated new TSK 0
I20260620 23:14:05.645617 24742 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:14:06.213424 24710 catalog_manager.cc:1259] Loaded cluster ID: 7fb168994d504fdea131ecb96d89a869
I20260620 23:14:06.213539 24710 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 04978d32ef074ba5b3f905538105d712: loading cluster ID for follower catalog manager: success
I20260620 23:14:06.216004 24710 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 04978d32ef074ba5b3f905538105d712: acquiring CA information for follower catalog manager: success
I20260620 23:14:06.217667 24710 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 04978d32ef074ba5b3f905538105d712: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:14:06.399861 24738 catalog_manager.cc:1259] Loaded cluster ID: 7fb168994d504fdea131ecb96d89a869
I20260620 23:14:06.400007 24738 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a8ad0769ea444ca6b4f8945bd0e9d51a: loading cluster ID for follower catalog manager: success
I20260620 23:14:06.402558 24738 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a8ad0769ea444ca6b4f8945bd0e9d51a: acquiring CA information for follower catalog manager: success
I20260620 23:14:06.404397 24738 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a8ad0769ea444ca6b4f8945bd0e9d51a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:14:06.790541 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 24471
I20260620 23:14:06.791889 24533 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:14:06.903753 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 24471
I20260620 23:14:06.924291 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 24545
I20260620 23:14:06.925446 24607 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:14:07.032068 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 24545
I20260620 23:14:07.052204 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 24619
I20260620 23:14:07.053381 24681 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:14:07.159793 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 24619
2026-06-20T23:14:07Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4789 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997042893763-11007-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:14:07 dist-test-slave-jkbq krb5kdc[24761](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:14:07 dist-test-slave-jkbq krb5kdc[24761](info): set up 2 sockets
Jun 20 23:14:07 dist-test-slave-jkbq krb5kdc[24761](info): commencing operation
krb5kdc: starting...
W20260620 23:14:08.251389 11007 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Jun 20 23:14:08 dist-test-slave-jkbq krb5kdc[24761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:14:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:14:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.191.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
I20260620 23:14:08.469954 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:39489
--webserver_interface=127.10.191.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:40315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:39489
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:14:08.789919 24774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:14:08.790251 24774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:14:08.790410 24774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:14:08.797655 24774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:14:08.797791 24774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:14:08.797885 24774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:14:08.797978 24774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:14:08.798053 24774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:14:08.807865 24774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:40315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:39489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:39489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:14:08.809226 24774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:14:08.810503 24774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:14:08.822045 24779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:08.821982 24780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:08.822023 24782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:14:08.822732 24774 server_base.cc:1034] running on GCE node
I20260620 23:14:08.823467 24774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:14:08.824762 24774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:14:08.825984 24774 hybrid_clock.cc:648] HybridClock initialized: now 1781997248825940 us; error 51 us; skew 500 ppm
I20260620 23:14:08.826308 24774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:14:08 dist-test-slave-jkbq krb5kdc[24761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997248, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:14:08.829277 24774 init.cc:377] Logged in from keytab as kudu/127.10.191.254@KRBTEST.COM (short username kudu)
I20260620 23:14:08.830631 24774 webserver.cc:460] Webserver started at http://127.10.191.254:37053/ using document root <none> and password file <none>
I20260620 23:14:08.831302 24774 fs_manager.cc:359] Metadata directory not provided
I20260620 23:14:08.831401 24774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:14:08.831650 24774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:14:08.834450 24774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "6e387c00ccd24e32b8145e54cfa1aa5e"
format_stamp: "Formatted at 2026-06-20 23:14:08 on dist-test-slave-jkbq"
server_key: "7e978ce804a340493e6913d1a8926dff"
server_key_iv: "0009ec1179899c5d802a948ea9e21050"
server_key_version: "encryptionkey@0"
I20260620 23:14:08.835204 24774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "6e387c00ccd24e32b8145e54cfa1aa5e"
format_stamp: "Formatted at 2026-06-20 23:14:08 on dist-test-slave-jkbq"
server_key: "7e978ce804a340493e6913d1a8926dff"
server_key_iv: "0009ec1179899c5d802a948ea9e21050"
server_key_version: "encryptionkey@0"
I20260620 23:14:08.840170 24774 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 23:14:08.843518 24789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:08.844794 24774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:14:08.844942 24774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "6e387c00ccd24e32b8145e54cfa1aa5e"
format_stamp: "Formatted at 2026-06-20 23:14:08 on dist-test-slave-jkbq"
server_key: "7e978ce804a340493e6913d1a8926dff"
server_key_iv: "0009ec1179899c5d802a948ea9e21050"
server_key_version: "encryptionkey@0"
I20260620 23:14:08.845110 24774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:14:08.874673 24774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:14:08.878597 24774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:14:08.878835 24774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:14:08.897480 24774 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:39489
I20260620 23:14:08.897508 24840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:39489 every 8 connection(s)
I20260620 23:14:08.898854 24774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:14:08.901037 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 24774
I20260620 23:14:08.901279 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:14:08.901950 11007 external_mini_cluster.cc:1428] Setting key 54bda6c22e896a63144339fb82b847d5
I20260620 23:14:08.902993 24841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 23:14:08 dist-test-slave-jkbq krb5kdc[24761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:14:08.918262 24841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e387c00ccd24e32b8145e54cfa1aa5e: Bootstrap starting.
I20260620 23:14:08.922453 24841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e387c00ccd24e32b8145e54cfa1aa5e: Neither blocks nor log segments found. Creating new log.
I20260620 23:14:08.923732 24841 log.cc:824] T 00000000000000000000000000000000 P 6e387c00ccd24e32b8145e54cfa1aa5e: Log is configured to *not* fsync() on all Append() calls
I20260620 23:14:08.926697 24841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e387c00ccd24e32b8145e54cfa1aa5e: No bootstrap required, opened a new log
I20260620 23:14:08.927275 24844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:08.903831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51628 (local address 127.10.191.254:39489)
0620 23:14:08.904636 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:08.904652 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:08.904701 (+ 49us) server_negotiation.cc:407] Connection header received
0620 23:14:08.906523 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:08.906551 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:08.907088 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:08.907538 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:08.909028 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:08.911117 (+ 2089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:08.912649 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:08.913726 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:08.913932 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:08.919089 (+ 5157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:08.919106 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:08.919126 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:08.919172 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:08.923170 (+ 3998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:08.923837 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:08.923846 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:08.923855 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:08.924073 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:08.924508 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:08.924513 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:08.924517 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:08.925090 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:08.925417 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:08.925843 (+ 426us) server_negotiation.cc:299] Negotiation successful
0620 23:14:08.926257 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":314,"threads_started":1}
I20260620 23:14:08.931340 11007 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:14:08.935483 24841 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e387c00ccd24e32b8145e54cfa1aa5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e387c00ccd24e32b8145e54cfa1aa5e" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39489 } }
I20260620 23:14:08.935834 24841 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e387c00ccd24e32b8145e54cfa1aa5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:14:08.935938 24841 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e387c00ccd24e32b8145e54cfa1aa5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e387c00ccd24e32b8145e54cfa1aa5e, State: Initialized, Role: FOLLOWER
I20260620 23:14:08.936473 24841 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e387c00ccd24e32b8145e54cfa1aa5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e387c00ccd24e32b8145e54cfa1aa5e" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39489 } }
I20260620 23:14:08.936699 24841 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e387c00ccd24e32b8145e54cfa1aa5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:14:08.936830 24841 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e387c00ccd24e32b8145e54cfa1aa5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:14:08.937006 24841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e387c00ccd24e32b8145e54cfa1aa5e [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:14:08.939113 24841 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e387c00ccd24e32b8145e54cfa1aa5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e387c00ccd24e32b8145e54cfa1aa5e" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39489 } }
I20260620 23:14:08.939591 24841 leader_election.cc:302] T 00000000000000000000000000000000 P 6e387c00ccd24e32b8145e54cfa1aa5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e387c00ccd24e32b8145e54cfa1aa5e; no voters:
I20260620 23:14:08.940160 24841 leader_election.cc:288] T 00000000000000000000000000000000 P 6e387c00ccd24e32b8145e54cfa1aa5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:14:08.940305 24846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e387c00ccd24e32b8145e54cfa1aa5e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:14:08.941033 24846 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e387c00ccd24e32b8145e54cfa1aa5e [term 1 LEADER]: Becoming Leader. State: Replica: 6e387c00ccd24e32b8145e54cfa1aa5e, State: Running, Role: LEADER
I20260620 23:14:08.941550 24846 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e387c00ccd24e32b8145e54cfa1aa5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e387c00ccd24e32b8145e54cfa1aa5e" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39489 } }
I20260620 23:14:08.942229 24841 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e387c00ccd24e32b8145e54cfa1aa5e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:14:08.944263 24847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e387c00ccd24e32b8145e54cfa1aa5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e387c00ccd24e32b8145e54cfa1aa5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e387c00ccd24e32b8145e54cfa1aa5e" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39489 } } }
I20260620 23:14:08.944690 24847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e387c00ccd24e32b8145e54cfa1aa5e [sys.catalog]: This master's current role is: LEADER
I20260620 23:14:08.945525 24848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e387c00ccd24e32b8145e54cfa1aa5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e387c00ccd24e32b8145e54cfa1aa5e. Latest consensus state: current_term: 1 leader_uuid: "6e387c00ccd24e32b8145e54cfa1aa5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e387c00ccd24e32b8145e54cfa1aa5e" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39489 } } }
I20260620 23:14:08.945876 24848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e387c00ccd24e32b8145e54cfa1aa5e [sys.catalog]: This master's current role is: LEADER
I20260620 23:14:08.947070 24854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:14:08.952304 24854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:14:08.960767 24854 catalog_manager.cc:1347] Generated new cluster ID: 90ccd223a5454dcd8eb78bba6a9d2033
I20260620 23:14:08.960928 24854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:14:08.983949 24854 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:14:08.985416 24854 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:14:08.992035 24854 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e387c00ccd24e32b8145e54cfa1aa5e: Generated new TSK 0
I20260620 23:14:08.992825 24854 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:14:09.044360 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 24774
I20260620 23:14:09.045514 24836 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:14:09.141224 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 24774
2026-06-20T23:14:09Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1977 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:14:09 dist-test-slave-jkbq krb5kdc[24869](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:14:09 dist-test-slave-jkbq krb5kdc[24869](info): set up 2 sockets
Jun 20 23:14:09 dist-test-slave-jkbq krb5kdc[24869](info): commencing operation
krb5kdc: starting...
W20260620 23:14:10.236330 11007 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Jun 20 23:14:10 dist-test-slave-jkbq krb5kdc[24869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:14:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:14:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.191.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
I20260620 23:14:10.474316 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:43585
--webserver_interface=127.10.191.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:39607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:43585
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:14:10.793241 24882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:14:10.793586 24882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:14:10.793742 24882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:14:10.801012 24882 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:14:10.801141 24882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:14:10.801237 24882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:14:10.801344 24882 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:14:10.801422 24882 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:14:10.811004 24882 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:39607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:43585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:43585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:14:10.812276 24882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:14:10.813606 24882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:14:10.825368 24890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:10.825448 24887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:10.825415 24888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:14:10.826285 24882 server_base.cc:1034] running on GCE node
I20260620 23:14:10.827083 24882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:14:10.828415 24882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:14:10.829617 24882 hybrid_clock.cc:648] HybridClock initialized: now 1781997250829571 us; error 30 us; skew 500 ppm
I20260620 23:14:10.829965 24882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:14:10 dist-test-slave-jkbq krb5kdc[24869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997250, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:14:10.833359 24882 init.cc:377] Logged in from keytab as kudu/127.10.191.254@KRBTEST.COM (short username kudu)
I20260620 23:14:10.834720 24882 webserver.cc:460] Webserver started at http://127.10.191.254:39291/ using document root <none> and password file <none>
I20260620 23:14:10.835424 24882 fs_manager.cc:359] Metadata directory not provided
I20260620 23:14:10.835529 24882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:14:10.835772 24882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:14:10.838630 24882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "5cc88259af6a46a7be7375becd6aef0f"
format_stamp: "Formatted at 2026-06-20 23:14:10 on dist-test-slave-jkbq"
server_key: "f23fe6202a7f8300f94db4d3a4b13825"
server_key_iv: "61f9b18d91e0633c9112f3abd4516173"
server_key_version: "encryptionkey@0"
I20260620 23:14:10.839426 24882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "5cc88259af6a46a7be7375becd6aef0f"
format_stamp: "Formatted at 2026-06-20 23:14:10 on dist-test-slave-jkbq"
server_key: "f23fe6202a7f8300f94db4d3a4b13825"
server_key_iv: "61f9b18d91e0633c9112f3abd4516173"
server_key_version: "encryptionkey@0"
I20260620 23:14:10.844264 24882 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 23:14:10.847608 24897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:10.848865 24882 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:14:10.849131 24882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "5cc88259af6a46a7be7375becd6aef0f"
format_stamp: "Formatted at 2026-06-20 23:14:10 on dist-test-slave-jkbq"
server_key: "f23fe6202a7f8300f94db4d3a4b13825"
server_key_iv: "61f9b18d91e0633c9112f3abd4516173"
server_key_version: "encryptionkey@0"
I20260620 23:14:10.849323 24882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:14:10.865970 24882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:14:10.870093 24882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:14:10.870318 24882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:14:10.888151 24882 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:43585
I20260620 23:14:10.888195 24948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:43585 every 8 connection(s)
I20260620 23:14:10.889524 24882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:14:10.893208 24949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:14:10.894500 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 24882
I20260620 23:14:10.894721 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:14:10.895439 11007 external_mini_cluster.cc:1428] Setting key d815cc0a0055a92ad3679ef98e9b120f
I20260620 23:14:10.907497 24949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f: Bootstrap starting.
Jun 20 23:14:10 dist-test-slave-jkbq krb5kdc[24869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:14:10.910492 24949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f: Neither blocks nor log segments found. Creating new log.
I20260620 23:14:10.911767 24949 log.cc:824] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f: Log is configured to *not* fsync() on all Append() calls
I20260620 23:14:10.915282 24949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f: No bootstrap required, opened a new log
I20260620 23:14:10.919739 24952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:10.897656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33744 (local address 127.10.191.254:43585)
0620 23:14:10.898478 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:10.898493 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:10.898555 (+ 62us) server_negotiation.cc:407] Connection header received
0620 23:14:10.900323 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:10.900383 (+ 60us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:10.900909 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:10.901316 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:10.902541 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:10.904607 (+ 2066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:10.905860 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:10.906858 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:10.907068 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:10.911166 (+ 4098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:10.911176 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:10.911194 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:10.911242 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:10.915179 (+ 3937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:10.915924 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:10.915936 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:10.915944 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:10.916157 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:10.916596 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:10.916601 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:10.916604 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:10.917343 (+ 739us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:10.917651 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:10.918022 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 23:14:10.918497 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":348,"threads_started":1}
I20260620 23:14:10.924607 24949 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc88259af6a46a7be7375becd6aef0f" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43585 } }
I20260620 23:14:10.925052 24949 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:14:10.925168 24949 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cc88259af6a46a7be7375becd6aef0f, State: Initialized, Role: FOLLOWER
I20260620 23:14:10.925714 24949 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc88259af6a46a7be7375becd6aef0f" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43585 } }
I20260620 23:14:10.925940 24949 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:14:10.926075 24949 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:14:10.926230 24949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:14:10.928426 24949 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc88259af6a46a7be7375becd6aef0f" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43585 } }
I20260620 23:14:10.928898 24949 leader_election.cc:302] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cc88259af6a46a7be7375becd6aef0f; no voters:
I20260620 23:14:10.929477 24949 leader_election.cc:288] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:14:10.929680 24954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:14:10.930593 24954 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [term 1 LEADER]: Becoming Leader. State: Replica: 5cc88259af6a46a7be7375becd6aef0f, State: Running, Role: LEADER
I20260620 23:14:10.931137 24954 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc88259af6a46a7be7375becd6aef0f" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43585 } }
I20260620 23:14:10.931597 24949 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:14:10.933759 24955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cc88259af6a46a7be7375becd6aef0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc88259af6a46a7be7375becd6aef0f" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43585 } } }
I20260620 23:14:10.934190 24955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [sys.catalog]: This master's current role is: LEADER
I20260620 23:14:10.934674 24956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cc88259af6a46a7be7375becd6aef0f. Latest consensus state: current_term: 1 leader_uuid: "5cc88259af6a46a7be7375becd6aef0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc88259af6a46a7be7375becd6aef0f" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43585 } } }
I20260620 23:14:10.935009 24956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [sys.catalog]: This master's current role is: LEADER
I20260620 23:14:10.935088 24963 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:14:10.941614 24963 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:14:10.950407 24963 catalog_manager.cc:1347] Generated new cluster ID: b8624621d19c4ecc95b33ee0bd1970fb
I20260620 23:14:10.950537 24963 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:14:10.966015 24963 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:14:10.967312 24963 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:14:10.975781 24963 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f: Generated new TSK 0
I20260620 23:14:10.976526 24963 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.191.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab.
Entry for principal kudu/127.10.191.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab.
I20260620 23:14:11.032940 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.193:0
--local_ip_for_outbound_sockets=127.10.191.193
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:43585
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:39607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:14:11.354049 24975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:14:11.354319 24975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:14:11.354470 24975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:14:11.361563 24975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:14:11.361670 24975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:14:11.361862 24975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.193
I20260620 23:14:11.371454 24975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:39607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:43585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.193
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:14:11.372720 24975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:14:11.374182 24975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:14:11.387287 24980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:11.387320 24983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:11.387300 24981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:14:11.388123 24975 server_base.cc:1034] running on GCE node
I20260620 23:14:11.388662 24975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:14:11.389523 24975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:14:11.390719 24975 hybrid_clock.cc:648] HybridClock initialized: now 1781997251390669 us; error 40 us; skew 500 ppm
I20260620 23:14:11.391068 24975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:14:11 dist-test-slave-jkbq krb5kdc[24869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997251, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:14:11.394207 24975 init.cc:377] Logged in from keytab as kudu/127.10.191.193@KRBTEST.COM (short username kudu)
I20260620 23:14:11.395696 24975 webserver.cc:460] Webserver started at http://127.10.191.193:37433/ using document root <none> and password file <none>
I20260620 23:14:11.396337 24975 fs_manager.cc:359] Metadata directory not provided
I20260620 23:14:11.396446 24975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:14:11.396696 24975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:14:11.399605 24975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/data/instance:
uuid: "2449fe0928b543e49e272470c621a01a"
format_stamp: "Formatted at 2026-06-20 23:14:11 on dist-test-slave-jkbq"
server_key: "0f6302b1228b49492a3c285bcecbdfc4"
server_key_iv: "310538ec03f0d7d0ccfa68a53f874241"
server_key_version: "encryptionkey@0"
I20260620 23:14:11.400313 24975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance:
uuid: "2449fe0928b543e49e272470c621a01a"
format_stamp: "Formatted at 2026-06-20 23:14:11 on dist-test-slave-jkbq"
server_key: "0f6302b1228b49492a3c285bcecbdfc4"
server_key_iv: "310538ec03f0d7d0ccfa68a53f874241"
server_key_version: "encryptionkey@0"
I20260620 23:14:11.405386 24975 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:14:11.408811 24990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:11.410149 24975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 23:14:11.410303 24975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/wal
uuid: "2449fe0928b543e49e272470c621a01a"
format_stamp: "Formatted at 2026-06-20 23:14:11 on dist-test-slave-jkbq"
server_key: "0f6302b1228b49492a3c285bcecbdfc4"
server_key_iv: "310538ec03f0d7d0ccfa68a53f874241"
server_key_version: "encryptionkey@0"
I20260620 23:14:11.410482 24975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:14:11.424374 24975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:14:11.428650 24975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:14:11.428944 24975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:14:11.429889 24975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:14:11.431335 24975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:14:11.431414 24975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:11.431571 24975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:14:11.431638 24975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:11.465659 24975 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.193:34215
I20260620 23:14:11.465701 25102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.193:34215 every 8 connection(s)
I20260620 23:14:11.466977 24975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
I20260620 23:14:11.477361 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 24975
I20260620 23:14:11.477605 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance
I20260620 23:14:11.478386 11007 external_mini_cluster.cc:1428] Setting key 2549289b08a1636300160271e4e1f5ee
Jun 20 23:14:11 dist-test-slave-jkbq krb5kdc[24869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997251, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:14:11.488139 25105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:11.469435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:58851 (local address 127.10.191.254:43585)
0620 23:14:11.469888 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:11.469892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:11.470664 (+ 772us) server_negotiation.cc:407] Connection header received
0620 23:14:11.472541 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:11.472546 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:11.472637 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:11.472750 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:11.474459 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:11.475405 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:11.477003 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:11.477391 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:11.477519 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:11.481677 (+ 4158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:11.481681 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:11.481685 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:11.481716 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:11.484405 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:11.485398 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:11.485403 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:11.485405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:11.485535 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:11.486025 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:11.486028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:11.486030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:11.486300 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:11.486427 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:11.487446 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0620 23:14:11.487639 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":250,"threads_started":1}
I20260620 23:14:11.489140 25106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:11.469815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.193:58851)
0620 23:14:11.470514 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:11.470551 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:11.472239 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:11.472985 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:11.472993 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:11.473673 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:14:11.474247 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:11.474264 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:11.475566 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:11.475570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:11.476811 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:11.476823 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:11.477617 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:11.477624 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:11.477927 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:11.478682 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:14:11.478705 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:14:11.481453 (+ 2748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:14:11.484564 (+ 3111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:11.484577 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:11.484598 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:11.485257 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:11.485650 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:11.485658 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:11.485665 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:11.485916 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:11.486432 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:14:11.486444 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:14:11.486981 (+ 537us) client_negotiation.cc:769] Sending connection context
0620 23:14:11.487363 (+ 382us) client_negotiation.cc:240] Negotiation successful
0620 23:14:11.487962 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":268,"threads_started":1}
I20260620 23:14:11.490700 25103 heartbeater.cc:347] Connected to a master server at 127.10.191.254:43585
I20260620 23:14:11.491060 25103 heartbeater.cc:464] Registering TS with master...
I20260620 23:14:11.491966 25103 heartbeater.cc:511] Master 127.10.191.254:43585 requested a full tablet report, sending...
I20260620 23:14:11.494158 24914 ts_manager.cc:194] Registered new tserver with Master: 2449fe0928b543e49e272470c621a01a (127.10.191.193:34215)
I20260620 23:14:11.495682 24914 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.193@KRBTEST.COM'} at 127.10.191.193:58851
WARNING: no policy specified for kudu/127.10.191.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab.
Entry for principal kudu/127.10.191.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab.
I20260620 23:14:11.530195 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.194:0
--local_ip_for_outbound_sockets=127.10.191.194
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:43585
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:39607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:14:11.852022 25109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:14:11.852355 25109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:14:11.852535 25109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:14:11.860200 25109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:14:11.860402 25109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:14:11.860606 25109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.194
I20260620 23:14:11.871222 25109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:39607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:43585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.194
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:14:11.872601 25109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:14:11.874012 25109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:14:11.887729 25115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:11.887943 25117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:11.887706 25114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:14:11.891852 25109 server_base.cc:1034] running on GCE node
I20260620 23:14:11.892529 25109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:14:11.893612 25109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:14:11.894881 25109 hybrid_clock.cc:648] HybridClock initialized: now 1781997251894794 us; error 68 us; skew 500 ppm
I20260620 23:14:11.895237 25109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:14:11 dist-test-slave-jkbq krb5kdc[24869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997251, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:14:11.899022 25109 init.cc:377] Logged in from keytab as kudu/127.10.191.194@KRBTEST.COM (short username kudu)
I20260620 23:14:11.900498 25109 webserver.cc:460] Webserver started at http://127.10.191.194:35681/ using document root <none> and password file <none>
I20260620 23:14:11.901178 25109 fs_manager.cc:359] Metadata directory not provided
I20260620 23:14:11.901304 25109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:14:11.901609 25109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:14:11.904609 25109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/data/instance:
uuid: "f9080c60e3e440999f4ed4fa4e342d8a"
format_stamp: "Formatted at 2026-06-20 23:14:11 on dist-test-slave-jkbq"
server_key: "f18e0ba4d25707f3a5b5713a61a69a62"
server_key_iv: "36314dcd557976ff2644feb0dc00c42d"
server_key_version: "encryptionkey@0"
I20260620 23:14:11.905385 25109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance:
uuid: "f9080c60e3e440999f4ed4fa4e342d8a"
format_stamp: "Formatted at 2026-06-20 23:14:11 on dist-test-slave-jkbq"
server_key: "f18e0ba4d25707f3a5b5713a61a69a62"
server_key_iv: "36314dcd557976ff2644feb0dc00c42d"
server_key_version: "encryptionkey@0"
I20260620 23:14:11.910741 25109 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 23:14:11.914196 25124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:11.915628 25109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:14:11.915804 25109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/wal
uuid: "f9080c60e3e440999f4ed4fa4e342d8a"
format_stamp: "Formatted at 2026-06-20 23:14:11 on dist-test-slave-jkbq"
server_key: "f18e0ba4d25707f3a5b5713a61a69a62"
server_key_iv: "36314dcd557976ff2644feb0dc00c42d"
server_key_version: "encryptionkey@0"
I20260620 23:14:11.916003 25109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:14:11.931838 25109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:14:11.936327 25109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:14:11.936616 25109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:14:11.937566 25109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:14:11.938848 25109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:14:11.939013 25109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:11.939164 25109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:14:11.939241 25109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:11.971789 25109 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.194:46539
I20260620 23:14:11.971887 25236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.194:46539 every 8 connection(s)
I20260620 23:14:11.973146 25109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
I20260620 23:14:11.982333 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 25109
I20260620 23:14:11.982537 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance
I20260620 23:14:11.983105 11007 external_mini_cluster.cc:1428] Setting key dba4218ef87d2dd98f9f5b104b8cb048
Jun 20 23:14:11 dist-test-slave-jkbq krb5kdc[24869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997251, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:14:11.992903 25105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:11.975642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:48309 (local address 127.10.191.254:43585)
0620 23:14:11.975850 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:11.975856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:11.977032 (+ 1176us) server_negotiation.cc:407] Connection header received
0620 23:14:11.978743 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:11.978748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:11.978816 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:11.978995 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:11.980486 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:11.981219 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:11.982979 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:11.983491 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:11.983632 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:11.987298 (+ 3666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:11.987303 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:11.987306 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:11.987337 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:11.989896 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:11.990628 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:11.990635 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:11.990638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:11.990773 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:11.991231 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:11.991237 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:11.991241 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:11.991504 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:11.991661 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:11.992326 (+ 665us) server_negotiation.cc:299] Negotiation successful
0620 23:14:11.992563 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 23:14:11.993870 25239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:11.976201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.194:48309)
0620 23:14:11.976880 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:11.976918 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:11.978504 (+ 1586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:11.979223 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:11.979231 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:11.979741 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:14:11.980305 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:11.980323 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:11.981379 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:11.981387 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:11.982778 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:11.982790 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:11.983718 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:11.983728 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:11.984022 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:11.984761 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:14:11.984795 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:14:11.987085 (+ 2290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:14:11.990064 (+ 2979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:11.990073 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:11.990089 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:11.990508 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:11.990907 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:11.990911 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:11.990915 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:11.991113 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:11.991633 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:14:11.991641 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:14:11.991972 (+ 331us) client_negotiation.cc:769] Sending connection context
0620 23:14:11.992229 (+ 257us) client_negotiation.cc:240] Negotiation successful
0620 23:14:11.992752 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":269,"threads_started":1}
I20260620 23:14:11.995391 25237 heartbeater.cc:347] Connected to a master server at 127.10.191.254:43585
I20260620 23:14:11.995712 25237 heartbeater.cc:464] Registering TS with master...
I20260620 23:14:11.996503 25237 heartbeater.cc:511] Master 127.10.191.254:43585 requested a full tablet report, sending...
I20260620 23:14:11.998009 24914 ts_manager.cc:194] Registered new tserver with Master: f9080c60e3e440999f4ed4fa4e342d8a (127.10.191.194:46539)
I20260620 23:14:11.999286 24914 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.194@KRBTEST.COM'} at 127.10.191.194:48309
WARNING: no policy specified for kudu/127.10.191.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab.
Entry for principal kudu/127.10.191.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab.
I20260620 23:14:12.032022 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.195:0
--local_ip_for_outbound_sockets=127.10.191.195
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:43585
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:39607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:14:12.354574 25242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:14:12.354830 25242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:14:12.354990 25242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:14:12.362202 25242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:14:12.362320 25242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:14:12.362433 25242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.195
I20260620 23:14:12.372098 25242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:39607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:43585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.195
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:14:12.373342 25242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:14:12.374603 25242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:14:12.387465 25248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:12.387508 25250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:12.387554 25247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:14:12.388355 25242 server_base.cc:1034] running on GCE node
I20260620 23:14:12.388877 25242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:14:12.389768 25242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:14:12.390990 25242 hybrid_clock.cc:648] HybridClock initialized: now 1781997252390947 us; error 64 us; skew 500 ppm
I20260620 23:14:12.391320 25242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:14:12 dist-test-slave-jkbq krb5kdc[24869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997252, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:14:12.399843 25242 init.cc:377] Logged in from keytab as kudu/127.10.191.195@KRBTEST.COM (short username kudu)
I20260620 23:14:12.401424 25242 webserver.cc:460] Webserver started at http://127.10.191.195:41539/ using document root <none> and password file <none>
I20260620 23:14:12.402148 25242 fs_manager.cc:359] Metadata directory not provided
I20260620 23:14:12.402293 25242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:14:12.402621 25242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:14:12.405639 25242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/data/instance:
uuid: "444f05061ce9490cb5725fc402922b2d"
format_stamp: "Formatted at 2026-06-20 23:14:12 on dist-test-slave-jkbq"
server_key: "08ea9c8bc3c44b03c0bc58960f40e04e"
server_key_iv: "3efdd9e69e74f58bfdffe17f4820d804"
server_key_version: "encryptionkey@0"
I20260620 23:14:12.406416 25242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance:
uuid: "444f05061ce9490cb5725fc402922b2d"
format_stamp: "Formatted at 2026-06-20 23:14:12 on dist-test-slave-jkbq"
server_key: "08ea9c8bc3c44b03c0bc58960f40e04e"
server_key_iv: "3efdd9e69e74f58bfdffe17f4820d804"
server_key_version: "encryptionkey@0"
I20260620 23:14:12.411643 25242 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 23:14:12.415735 25257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:12.417085 25242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:14:12.417255 25242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/wal
uuid: "444f05061ce9490cb5725fc402922b2d"
format_stamp: "Formatted at 2026-06-20 23:14:12 on dist-test-slave-jkbq"
server_key: "08ea9c8bc3c44b03c0bc58960f40e04e"
server_key_iv: "3efdd9e69e74f58bfdffe17f4820d804"
server_key_version: "encryptionkey@0"
I20260620 23:14:12.417459 25242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:14:12.441599 25242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:14:12.445640 25242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:14:12.445930 25242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:14:12.446962 25242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:14:12.448283 25242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:14:12.448387 25242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:12.448539 25242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:14:12.448612 25242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:12.483911 25242 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.195:46283
I20260620 23:14:12.483994 25369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.195:46283 every 8 connection(s)
I20260620 23:14:12.485188 25242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
I20260620 23:14:12.493649 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 25242
I20260620 23:14:12.493821 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance
I20260620 23:14:12.494288 11007 external_mini_cluster.cc:1428] Setting key 22c0b6a1e9ee6129ea9672bc256aca64
Jun 20 23:14:12 dist-test-slave-jkbq krb5kdc[24869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997252, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:14:12.499540 25103 heartbeater.cc:503] Master 127.10.191.254:43585 was elected leader, sending a full tablet report...
I20260620 23:14:12.505250 25105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:12.487835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:45591 (local address 127.10.191.254:43585)
0620 23:14:12.488008 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:12.488012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:12.489081 (+ 1069us) server_negotiation.cc:407] Connection header received
0620 23:14:12.490902 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:12.490909 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:12.491011 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:12.491127 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:12.492580 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:12.493434 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:12.495223 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:12.495809 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:12.495953 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:12.499515 (+ 3562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:12.499524 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:12.499530 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:12.499572 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:12.502412 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:12.503125 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:12.503128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:12.503131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:12.503247 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:12.503805 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:12.503808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:12.503810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:12.504096 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:12.504231 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:12.504875 (+ 644us) server_negotiation.cc:299] Negotiation successful
0620 23:14:12.505020 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 23:14:12.506224 25372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:12.488256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.195:45591)
0620 23:14:12.488920 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:12.488961 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:12.490611 (+ 1650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:12.491368 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:12.491376 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:12.491876 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:14:12.492416 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:12.492433 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:12.493626 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:12.493630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:12.495098 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:12.495110 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:12.496013 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:12.496018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:12.496307 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:12.497034 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:14:12.497064 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:14:12.499303 (+ 2239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:14:12.502541 (+ 3238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:12.502547 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:12.502557 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:12.503024 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:12.503412 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:12.503416 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:12.503418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:12.503708 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:12.504248 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:14:12.504256 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:14:12.504572 (+ 316us) client_negotiation.cc:769] Sending connection context
0620 23:14:12.504785 (+ 213us) client_negotiation.cc:240] Negotiation successful
0620 23:14:12.505132 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":258,"threads_started":1}
I20260620 23:14:12.507560 25370 heartbeater.cc:347] Connected to a master server at 127.10.191.254:43585
I20260620 23:14:12.507862 25370 heartbeater.cc:464] Registering TS with master...
I20260620 23:14:12.508685 25370 heartbeater.cc:511] Master 127.10.191.254:43585 requested a full tablet report, sending...
I20260620 23:14:12.510046 24914 ts_manager.cc:194] Registered new tserver with Master: 444f05061ce9490cb5725fc402922b2d (127.10.191.195:46283)
I20260620 23:14:12.510848 24914 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.195@KRBTEST.COM'} at 127.10.191.195:45591
I20260620 23:14:12.521788 11007 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:14:12.538681 25105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:12.526091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33750 (local address 127.10.191.254:43585)
0620 23:14:12.526281 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:12.526285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:12.526757 (+ 472us) server_negotiation.cc:407] Connection header received
0620 23:14:12.526981 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:12.526985 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:12.527047 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:12.527142 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:12.528506 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:12.529217 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:12.530495 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:12.530979 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:12.531079 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:12.533453 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:12.533458 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:12.533461 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:12.533498 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:12.536125 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:12.536806 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:12.536813 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:12.536816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:12.536943 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:12.537365 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:12.537369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:12.537372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:12.537675 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:12.537871 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:12.538279 (+ 408us) server_negotiation.cc:299] Negotiation successful
0620 23:14:12.538451 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 23:14:12.542716 24914 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:14:12.549619 24914 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:14:12.573807 25382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:12.561508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.194:46539 (local address 127.0.0.1:47948)
0620 23:14:12.562213 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:12.562266 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:12.562418 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:12.563214 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:12.563222 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:12.563254 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:14:12.563710 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:12.563729 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:12.566585 (+ 2856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:12.566594 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:12.569781 (+ 3187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:12.569804 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:12.572441 (+ 2637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:12.572450 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:12.572785 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:12.572880 (+ 95us) client_negotiation.cc:769] Sending connection context
0620 23:14:12.573015 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 23:14:12.573273 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":289,"threads_started":1}
I20260620 23:14:12.575457 25383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:12.561701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47948 (local address 127.10.191.194:46539)
0620 23:14:12.562300 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:12.562309 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:12.562345 (+ 36us) server_negotiation.cc:407] Connection header received
0620 23:14:12.562639 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:12.562646 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:12.562960 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:12.563987 (+ 1027us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:14:12.564263 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:12.566381 (+ 2118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:12.570081 (+ 3700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:12.572159 (+ 2078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:12.574474 (+ 2315us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:12.574729 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:12.574843 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 23:14:12.575098 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":310,"threads_started":1}
I20260620 23:14:12.580148 25172 tablet_service.cc:1467] Processing CreateTablet for tablet 16c60d1376eb4a2dbb7325a1633f94cb (DEFAULT_TABLE table=test-table [id=8fd887122364412ba23579790beeb581]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:14:12.581604 25172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16c60d1376eb4a2dbb7325a1633f94cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:14:12.583447 25382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:12.563252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:34215 (local address 127.0.0.1:41542)
0620 23:14:12.574095 (+ 10843us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:12.574113 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:12.574238 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:12.576822 (+ 2584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:12.576830 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:12.576870 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:14:12.577093 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:12.577107 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:12.578983 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:12.578988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:12.580845 (+ 1857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:12.580857 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:12.582598 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:12.582603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:12.582797 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:12.582844 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 23:14:12.582971 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 23:14:12.583123 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10756,"thread_start_us":11973,"threads_started":1}
I20260620 23:14:12.583537 25387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:12.563895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41542 (local address 127.10.191.193:34215)
0620 23:14:12.576142 (+ 12247us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:12.576153 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:12.576183 (+ 30us) server_negotiation.cc:407] Connection header received
0620 23:14:12.576286 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:12.576294 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:12.576578 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:12.576790 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:14:12.577341 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:12.578791 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:12.581037 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:12.582419 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:12.582611 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:12.582766 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:12.583009 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 23:14:12.583165 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12072,"thread_start_us":11886,"threads_started":1}
I20260620 23:14:12.586287 25384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:12.562677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.195:46283 (local address 127.0.0.1:56536)
0620 23:14:12.563209 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:12.563226 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:12.563353 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:12.578211 (+ 14858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:12.578217 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:12.578243 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:14:12.578576 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:12.578587 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:12.580549 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:12.580558 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:12.583411 (+ 2853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:12.583426 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:12.585507 (+ 2081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:12.585515 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:12.585687 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:12.585718 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 23:14:12.585816 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 23:14:12.585952 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":269,"threads_started":1}
I20260620 23:14:12.586364 25385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:12.563031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56536 (local address 127.10.191.195:46283)
0620 23:14:12.577625 (+ 14594us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:12.577633 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:12.577664 (+ 31us) server_negotiation.cc:407] Connection header received
0620 23:14:12.577761 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:12.577768 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:12.578015 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:12.578200 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:14:12.578769 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:12.580400 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:12.583927 (+ 3527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:12.585296 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:12.585490 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:12.585646 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:12.585873 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 23:14:12.586036 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14413,"thread_start_us":14248,"threads_started":1}
I20260620 23:14:12.587240 25038 tablet_service.cc:1467] Processing CreateTablet for tablet 16c60d1376eb4a2dbb7325a1633f94cb (DEFAULT_TABLE table=test-table [id=8fd887122364412ba23579790beeb581]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:14:12.588661 25038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16c60d1376eb4a2dbb7325a1633f94cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:14:12.590108 25305 tablet_service.cc:1467] Processing CreateTablet for tablet 16c60d1376eb4a2dbb7325a1633f94cb (DEFAULT_TABLE table=test-table [id=8fd887122364412ba23579790beeb581]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:14:12.591514 25305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16c60d1376eb4a2dbb7325a1633f94cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:14:12.596429 25388 tablet_bootstrap.cc:492] T 16c60d1376eb4a2dbb7325a1633f94cb P f9080c60e3e440999f4ed4fa4e342d8a: Bootstrap starting.
I20260620 23:14:12.600144 25388 tablet_bootstrap.cc:654] T 16c60d1376eb4a2dbb7325a1633f94cb P f9080c60e3e440999f4ed4fa4e342d8a: Neither blocks nor log segments found. Creating new log.
I20260620 23:14:12.600965 25389 tablet_bootstrap.cc:492] T 16c60d1376eb4a2dbb7325a1633f94cb P 2449fe0928b543e49e272470c621a01a: Bootstrap starting.
I20260620 23:14:12.601508 25388 log.cc:824] T 16c60d1376eb4a2dbb7325a1633f94cb P f9080c60e3e440999f4ed4fa4e342d8a: Log is configured to *not* fsync() on all Append() calls
I20260620 23:14:12.604691 25389 tablet_bootstrap.cc:654] T 16c60d1376eb4a2dbb7325a1633f94cb P 2449fe0928b543e49e272470c621a01a: Neither blocks nor log segments found. Creating new log.
I20260620 23:14:12.605381 25388 tablet_bootstrap.cc:492] T 16c60d1376eb4a2dbb7325a1633f94cb P f9080c60e3e440999f4ed4fa4e342d8a: No bootstrap required, opened a new log
I20260620 23:14:12.605659 25388 ts_tablet_manager.cc:1397] T 16c60d1376eb4a2dbb7325a1633f94cb P f9080c60e3e440999f4ed4fa4e342d8a: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260620 23:14:12.606257 25389 log.cc:824] T 16c60d1376eb4a2dbb7325a1633f94cb P 2449fe0928b543e49e272470c621a01a: Log is configured to *not* fsync() on all Append() calls
I20260620 23:14:12.611752 25391 tablet_bootstrap.cc:492] T 16c60d1376eb4a2dbb7325a1633f94cb P 444f05061ce9490cb5725fc402922b2d: Bootstrap starting.
I20260620 23:14:12.612465 25389 tablet_bootstrap.cc:492] T 16c60d1376eb4a2dbb7325a1633f94cb P 2449fe0928b543e49e272470c621a01a: No bootstrap required, opened a new log
I20260620 23:14:12.612737 25389 ts_tablet_manager.cc:1397] T 16c60d1376eb4a2dbb7325a1633f94cb P 2449fe0928b543e49e272470c621a01a: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260620 23:14:12.614177 25388 raft_consensus.cc:348] T 16c60d1376eb4a2dbb7325a1633f94cb P f9080c60e3e440999f4ed4fa4e342d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9080c60e3e440999f4ed4fa4e342d8a" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 46539 } } peers { permanent_uuid: "444f05061ce9490cb5725fc402922b2d" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46283 } } peers { permanent_uuid: "2449fe0928b543e49e272470c621a01a" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 34215 } }
I20260620 23:14:12.614683 25388 raft_consensus.cc:374] T 16c60d1376eb4a2dbb7325a1633f94cb P f9080c60e3e440999f4ed4fa4e342d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:14:12.614789 25388 raft_consensus.cc:729] T 16c60d1376eb4a2dbb7325a1633f94cb P f9080c60e3e440999f4ed4fa4e342d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9080c60e3e440999f4ed4fa4e342d8a, State: Initialized, Role: FOLLOWER
I20260620 23:14:12.615428 25388 consensus_queue.cc:260] T 16c60d1376eb4a2dbb7325a1633f94cb P f9080c60e3e440999f4ed4fa4e342d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9080c60e3e440999f4ed4fa4e342d8a" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 46539 } } peers { permanent_uuid: "444f05061ce9490cb5725fc402922b2d" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46283 } } peers { permanent_uuid: "2449fe0928b543e49e272470c621a01a" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 34215 } }
I20260620 23:14:12.616500 25391 tablet_bootstrap.cc:654] T 16c60d1376eb4a2dbb7325a1633f94cb P 444f05061ce9490cb5725fc402922b2d: Neither blocks nor log segments found. Creating new log.
I20260620 23:14:12.618084 25391 log.cc:824] T 16c60d1376eb4a2dbb7325a1633f94cb P 444f05061ce9490cb5725fc402922b2d: Log is configured to *not* fsync() on all Append() calls
I20260620 23:14:12.619689 25237 heartbeater.cc:503] Master 127.10.191.254:43585 was elected leader, sending a full tablet report...
I20260620 23:14:12.620394 25388 ts_tablet_manager.cc:1428] T 16c60d1376eb4a2dbb7325a1633f94cb P f9080c60e3e440999f4ed4fa4e342d8a: Time spent starting tablet: real 0.015s user 0.008s sys 0.008s
I20260620 23:14:12.621460 25391 tablet_bootstrap.cc:492] T 16c60d1376eb4a2dbb7325a1633f94cb P 444f05061ce9490cb5725fc402922b2d: No bootstrap required, opened a new log
I20260620 23:14:12.621729 25391 ts_tablet_manager.cc:1397] T 16c60d1376eb4a2dbb7325a1633f94cb P 444f05061ce9490cb5725fc402922b2d: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260620 23:14:12.622646 25389 raft_consensus.cc:348] T 16c60d1376eb4a2dbb7325a1633f94cb P 2449fe0928b543e49e272470c621a01a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9080c60e3e440999f4ed4fa4e342d8a" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 46539 } } peers { permanent_uuid: "444f05061ce9490cb5725fc402922b2d" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46283 } } peers { permanent_uuid: "2449fe0928b543e49e272470c621a01a" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 34215 } }
I20260620 23:14:12.623131 25389 raft_consensus.cc:374] T 16c60d1376eb4a2dbb7325a1633f94cb P 2449fe0928b543e49e272470c621a01a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:14:12.623260 25389 raft_consensus.cc:729] T 16c60d1376eb4a2dbb7325a1633f94cb P 2449fe0928b543e49e272470c621a01a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2449fe0928b543e49e272470c621a01a, State: Initialized, Role: FOLLOWER
I20260620 23:14:12.623946 25389 consensus_queue.cc:260] T 16c60d1376eb4a2dbb7325a1633f94cb P 2449fe0928b543e49e272470c621a01a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9080c60e3e440999f4ed4fa4e342d8a" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 46539 } } peers { permanent_uuid: "444f05061ce9490cb5725fc402922b2d" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46283 } } peers { permanent_uuid: "2449fe0928b543e49e272470c621a01a" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 34215 } }
I20260620 23:14:12.625931 25389 ts_tablet_manager.cc:1428] T 16c60d1376eb4a2dbb7325a1633f94cb P 2449fe0928b543e49e272470c621a01a: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
I20260620 23:14:12.631337 25391 raft_consensus.cc:348] T 16c60d1376eb4a2dbb7325a1633f94cb P 444f05061ce9490cb5725fc402922b2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9080c60e3e440999f4ed4fa4e342d8a" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 46539 } } peers { permanent_uuid: "444f05061ce9490cb5725fc402922b2d" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46283 } } peers { permanent_uuid: "2449fe0928b543e49e272470c621a01a" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 34215 } }
I20260620 23:14:12.631698 25391 raft_consensus.cc:374] T 16c60d1376eb4a2dbb7325a1633f94cb P 444f05061ce9490cb5725fc402922b2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:14:12.631803 25391 raft_consensus.cc:729] T 16c60d1376eb4a2dbb7325a1633f94cb P 444f05061ce9490cb5725fc402922b2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 444f05061ce9490cb5725fc402922b2d, State: Initialized, Role: FOLLOWER
I20260620 23:14:12.632365 25391 consensus_queue.cc:260] T 16c60d1376eb4a2dbb7325a1633f94cb P 444f05061ce9490cb5725fc402922b2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9080c60e3e440999f4ed4fa4e342d8a" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 46539 } } peers { permanent_uuid: "444f05061ce9490cb5725fc402922b2d" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46283 } } peers { permanent_uuid: "2449fe0928b543e49e272470c621a01a" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 34215 } }
I20260620 23:14:12.633302 25370 heartbeater.cc:503] Master 127.10.191.254:43585 was elected leader, sending a full tablet report...
I20260620 23:14:12.633751 25391 ts_tablet_manager.cc:1428] T 16c60d1376eb4a2dbb7325a1633f94cb P 444f05061ce9490cb5725fc402922b2d: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260620 23:14:12.641916 25393 raft_consensus.cc:482] T 16c60d1376eb4a2dbb7325a1633f94cb P f9080c60e3e440999f4ed4fa4e342d8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:14:12.642179 25393 raft_consensus.cc:504] T 16c60d1376eb4a2dbb7325a1633f94cb P f9080c60e3e440999f4ed4fa4e342d8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9080c60e3e440999f4ed4fa4e342d8a" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 46539 } } peers { permanent_uuid: "444f05061ce9490cb5725fc402922b2d" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46283 } } peers { permanent_uuid: "2449fe0928b543e49e272470c621a01a" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 34215 } }
I20260620 23:14:12.643867 25393 leader_election.cc:288] T 16c60d1376eb4a2dbb7325a1633f94cb P f9080c60e3e440999f4ed4fa4e342d8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 444f05061ce9490cb5725fc402922b2d (127.10.191.195:46283), 2449fe0928b543e49e272470c621a01a (127.10.191.193:34215)
I20260620 23:14:12.650943 25385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:12.644205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:48711 (local address 127.10.191.195:46283)
0620 23:14:12.644406 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:12.644410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:12.645078 (+ 668us) server_negotiation.cc:407] Connection header received
0620 23:14:12.645302 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:12.645311 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:12.645409 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:12.645473 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:14:12.646287 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:12.647253 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:12.649093 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:12.649913 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:12.650030 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:12.650079 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:12.650471 (+ 392us) server_negotiation.cc:299] Negotiation successful
0620 23:14:12.650608 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 23:14:12.650972 25387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:12.645400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:55759 (local address 127.10.191.193:34215)
0620 23:14:12.645659 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:12.645665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:12.645685 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:14:12.645890 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:12.645904 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:12.645975 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:12.646075 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:14:12.646707 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:12.647537 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:12.649245 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:12.650041 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:12.650193 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:12.650263 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:12.650578 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 23:14:12.650720 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260620 23:14:12.651152 25397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:12.644214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.195:46283 (local address 127.10.191.194:48711)
0620 23:14:12.644951 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:12.644972 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:12.645104 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:12.645694 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:12.645701 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:12.645758 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:14:12.646133 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:12.646146 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:12.647378 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:12.647384 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:12.648940 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:12.648952 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:12.650122 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:12.650131 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:12.650317 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:12.650344 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 23:14:12.650424 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 23:14:12.650536 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"spinlock_wait_cycles":6016,"thread_start_us":391,"threads_started":1}
I20260620 23:14:12.651233 25398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:12.645067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:34215 (local address 127.10.191.194:55759)
0620 23:14:12.645619 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:12.645637 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:12.645743 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:12.646094 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:12.646097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:12.646119 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:14:12.646556 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:12.646572 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:12.647679 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:12.647683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:12.649093 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:12.649107 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:12.650247 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:12.650251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:12.650388 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:12.650447 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 23:14:12.650537 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 23:14:12.650653 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"spinlock_wait_cycles":3456,"thread_start_us":278,"threads_started":1}
I20260620 23:14:12.652109 25325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16c60d1376eb4a2dbb7325a1633f94cb" candidate_uuid: "f9080c60e3e440999f4ed4fa4e342d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "444f05061ce9490cb5725fc402922b2d" is_pre_election: true
I20260620 23:14:12.652379 25058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16c60d1376eb4a2dbb7325a1633f94cb" candidate_uuid: "f9080c60e3e440999f4ed4fa4e342d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2449fe0928b543e49e272470c621a01a" is_pre_election: true
I20260620 23:14:12.652706 25325 raft_consensus.cc:2454] T 16c60d1376eb4a2dbb7325a1633f94cb P 444f05061ce9490cb5725fc402922b2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9080c60e3e440999f4ed4fa4e342d8a in term 0.
I20260620 23:14:12.652918 25058 raft_consensus.cc:2454] T 16c60d1376eb4a2dbb7325a1633f94cb P 2449fe0928b543e49e272470c621a01a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9080c60e3e440999f4ed4fa4e342d8a in term 0.
I20260620 23:14:12.653421 25126 leader_election.cc:302] T 16c60d1376eb4a2dbb7325a1633f94cb P f9080c60e3e440999f4ed4fa4e342d8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 444f05061ce9490cb5725fc402922b2d, f9080c60e3e440999f4ed4fa4e342d8a; no voters:
I20260620 23:14:12.653836 25393 raft_consensus.cc:2781] T 16c60d1376eb4a2dbb7325a1633f94cb P f9080c60e3e440999f4ed4fa4e342d8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:14:12.653992 25393 raft_consensus.cc:482] T 16c60d1376eb4a2dbb7325a1633f94cb P f9080c60e3e440999f4ed4fa4e342d8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:14:12.654104 25393 raft_consensus.cc:3037] T 16c60d1376eb4a2dbb7325a1633f94cb P f9080c60e3e440999f4ed4fa4e342d8a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:14:12.656533 25393 raft_consensus.cc:504] T 16c60d1376eb4a2dbb7325a1633f94cb P f9080c60e3e440999f4ed4fa4e342d8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9080c60e3e440999f4ed4fa4e342d8a" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 46539 } } peers { permanent_uuid: "444f05061ce9490cb5725fc402922b2d" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46283 } } peers { permanent_uuid: "2449fe0928b543e49e272470c621a01a" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 34215 } }
I20260620 23:14:12.657125 25393 leader_election.cc:288] T 16c60d1376eb4a2dbb7325a1633f94cb P f9080c60e3e440999f4ed4fa4e342d8a [CANDIDATE]: Term 1 election: Requested vote from peers 444f05061ce9490cb5725fc402922b2d (127.10.191.195:46283), 2449fe0928b543e49e272470c621a01a (127.10.191.193:34215)
I20260620 23:14:12.657502 25325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16c60d1376eb4a2dbb7325a1633f94cb" candidate_uuid: "f9080c60e3e440999f4ed4fa4e342d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "444f05061ce9490cb5725fc402922b2d"
I20260620 23:14:12.657575 25058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16c60d1376eb4a2dbb7325a1633f94cb" candidate_uuid: "f9080c60e3e440999f4ed4fa4e342d8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2449fe0928b543e49e272470c621a01a"
I20260620 23:14:12.657699 25325 raft_consensus.cc:3037] T 16c60d1376eb4a2dbb7325a1633f94cb P 444f05061ce9490cb5725fc402922b2d [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:14:12.657814 25058 raft_consensus.cc:3037] T 16c60d1376eb4a2dbb7325a1633f94cb P 2449fe0928b543e49e272470c621a01a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:14:12.659986 25325 raft_consensus.cc:2454] T 16c60d1376eb4a2dbb7325a1633f94cb P 444f05061ce9490cb5725fc402922b2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9080c60e3e440999f4ed4fa4e342d8a in term 1.
I20260620 23:14:12.659998 25058 raft_consensus.cc:2454] T 16c60d1376eb4a2dbb7325a1633f94cb P 2449fe0928b543e49e272470c621a01a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9080c60e3e440999f4ed4fa4e342d8a in term 1.
I20260620 23:14:12.660357 25126 leader_election.cc:302] T 16c60d1376eb4a2dbb7325a1633f94cb P f9080c60e3e440999f4ed4fa4e342d8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2449fe0928b543e49e272470c621a01a, f9080c60e3e440999f4ed4fa4e342d8a; no voters:
I20260620 23:14:12.660676 25393 raft_consensus.cc:2781] T 16c60d1376eb4a2dbb7325a1633f94cb P f9080c60e3e440999f4ed4fa4e342d8a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:14:12.661118 25393 raft_consensus.cc:686] T 16c60d1376eb4a2dbb7325a1633f94cb P f9080c60e3e440999f4ed4fa4e342d8a [term 1 LEADER]: Becoming Leader. State: Replica: f9080c60e3e440999f4ed4fa4e342d8a, State: Running, Role: LEADER
I20260620 23:14:12.661578 25393 consensus_queue.cc:237] T 16c60d1376eb4a2dbb7325a1633f94cb P f9080c60e3e440999f4ed4fa4e342d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9080c60e3e440999f4ed4fa4e342d8a" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 46539 } } peers { permanent_uuid: "444f05061ce9490cb5725fc402922b2d" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46283 } } peers { permanent_uuid: "2449fe0928b543e49e272470c621a01a" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 34215 } }
I20260620 23:14:12.665864 24913 catalog_manager.cc:5495] T 16c60d1376eb4a2dbb7325a1633f94cb P f9080c60e3e440999f4ed4fa4e342d8a reported cstate change: term changed from 0 to 1, leader changed from <none> to f9080c60e3e440999f4ed4fa4e342d8a (127.10.191.194). New cstate: current_term: 1 leader_uuid: "f9080c60e3e440999f4ed4fa4e342d8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9080c60e3e440999f4ed4fa4e342d8a" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 46539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "444f05061ce9490cb5725fc402922b2d" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2449fe0928b543e49e272470c621a01a" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 34215 } health_report { overall_health: UNKNOWN } } }
I20260620 23:14:12.686950 25383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:12.681591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47952 (local address 127.10.191.194:46539)
0620 23:14:12.681795 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:12.681800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:12.681818 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:14:12.682012 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:12.682016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:12.682089 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:12.682235 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:14:12.682637 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:12.683390 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:12.684810 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:12.685206 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:12.685339 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:12.685842 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:14:12.686315 (+ 473us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:14:12.686485 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:12.686545 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 23:14:12.686617 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 23:14:12.694731 25058 raft_consensus.cc:1261] T 16c60d1376eb4a2dbb7325a1633f94cb P 2449fe0928b543e49e272470c621a01a [term 1 FOLLOWER]: Refusing update from remote peer f9080c60e3e440999f4ed4fa4e342d8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:14:12.694744 25325 raft_consensus.cc:1261] T 16c60d1376eb4a2dbb7325a1633f94cb P 444f05061ce9490cb5725fc402922b2d [term 1 FOLLOWER]: Refusing update from remote peer f9080c60e3e440999f4ed4fa4e342d8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:14:12.695676 25399 consensus_queue.cc:1035] T 16c60d1376eb4a2dbb7325a1633f94cb P f9080c60e3e440999f4ed4fa4e342d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "444f05061ce9490cb5725fc402922b2d" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:14:12.696005 25393 consensus_queue.cc:1035] T 16c60d1376eb4a2dbb7325a1633f94cb P f9080c60e3e440999f4ed4fa4e342d8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2449fe0928b543e49e272470c621a01a" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 34215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:14:12.707669 25404 mvcc.cc:205] Tried to move back new op lower bound from 7299060747028369408 to 7299060746905665536. Current Snapshot: MvccSnapshot[applied={T|T < 7299060747028369408}]
I20260620 23:14:12.707106 25402 mvcc.cc:205] Tried to move back new op lower bound from 7299060747028369408 to 7299060746905665536. Current Snapshot: MvccSnapshot[applied={T|T < 7299060747028369408}]
I20260620 23:14:12.720618 24914 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:14:12.720916 24914 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:14:12.724570 24914 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f: Sending DeleteTablet for 3 replicas of tablet 16c60d1376eb4a2dbb7325a1633f94cb
I20260620 23:14:12.725931 25038 tablet_service.cc:1514] Processing DeleteTablet for tablet 16c60d1376eb4a2dbb7325a1633f94cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:14:12 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:41542
I20260620 23:14:12.726082 25172 tablet_service.cc:1514] Processing DeleteTablet for tablet 16c60d1376eb4a2dbb7325a1633f94cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:14:12 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:47948
I20260620 23:14:12.726886 25411 tablet_replica.cc:330] T 16c60d1376eb4a2dbb7325a1633f94cb P 2449fe0928b543e49e272470c621a01a: stopping tablet replica
I20260620 23:14:12.726997 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 24975
I20260620 23:14:12.726835 25305 tablet_service.cc:1514] Processing DeleteTablet for tablet 16c60d1376eb4a2dbb7325a1633f94cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:14:12 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:56536
I20260620 23:14:12.727321 25411 raft_consensus.cc:2229] T 16c60d1376eb4a2dbb7325a1633f94cb P 2449fe0928b543e49e272470c621a01a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:14:12.727806 25411 raft_consensus.cc:2258] T 16c60d1376eb4a2dbb7325a1633f94cb P 2449fe0928b543e49e272470c621a01a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:14:12.727859 25412 tablet_replica.cc:330] T 16c60d1376eb4a2dbb7325a1633f94cb P 444f05061ce9490cb5725fc402922b2d: stopping tablet replica
I20260620 23:14:12.728482 25412 raft_consensus.cc:2229] T 16c60d1376eb4a2dbb7325a1633f94cb P 444f05061ce9490cb5725fc402922b2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:14:12.729071 25412 raft_consensus.cc:2258] T 16c60d1376eb4a2dbb7325a1633f94cb P 444f05061ce9490cb5725fc402922b2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:14:12.731702 25412 ts_tablet_manager.cc:1905] T 16c60d1376eb4a2dbb7325a1633f94cb P 444f05061ce9490cb5725fc402922b2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:14:12.732887 25411 ts_tablet_manager.cc:1905] T 16c60d1376eb4a2dbb7325a1633f94cb P 2449fe0928b543e49e272470c621a01a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:14:12.739454 25412 ts_tablet_manager.cc:1918] T 16c60d1376eb4a2dbb7325a1633f94cb P 444f05061ce9490cb5725fc402922b2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:14:12.739670 25412 log.cc:1196] T 16c60d1376eb4a2dbb7325a1633f94cb P 444f05061ce9490cb5725fc402922b2d: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/wal/wals/16c60d1376eb4a2dbb7325a1633f94cb
I20260620 23:14:12.740170 25411 ts_tablet_manager.cc:1918] T 16c60d1376eb4a2dbb7325a1633f94cb P 2449fe0928b543e49e272470c621a01a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:14:12.740310 25412 ts_tablet_manager.cc:1939] T 16c60d1376eb4a2dbb7325a1633f94cb P 444f05061ce9490cb5725fc402922b2d: Deleting consensus metadata
I20260620 23:14:12.740327 25411 log.cc:1196] T 16c60d1376eb4a2dbb7325a1633f94cb P 2449fe0928b543e49e272470c621a01a: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/wal/wals/16c60d1376eb4a2dbb7325a1633f94cb
I20260620 23:14:12.740878 25411 ts_tablet_manager.cc:1939] T 16c60d1376eb4a2dbb7325a1633f94cb P 2449fe0928b543e49e272470c621a01a: Deleting consensus metadata
I20260620 23:14:12.742614 24899 catalog_manager.cc:4841] TS 444f05061ce9490cb5725fc402922b2d (127.10.191.195:46283): tablet 16c60d1376eb4a2dbb7325a1633f94cb (table test-table [id=8fd887122364412ba23579790beeb581]) successfully deleted
I20260620 23:14:12.743005 24899 catalog_manager.cc:4841] TS 2449fe0928b543e49e272470c621a01a (127.10.191.193:34215): tablet 16c60d1376eb4a2dbb7325a1633f94cb (table test-table [id=8fd887122364412ba23579790beeb581]) successfully deleted
Jun 20 23:14:12 dist-test-slave-jkbq krb5kdc[24869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.193@KRBTEST.COM
I20260620 23:14:12.747424 25413 tablet_replica.cc:330] T 16c60d1376eb4a2dbb7325a1633f94cb P f9080c60e3e440999f4ed4fa4e342d8a: stopping tablet replica
I20260620 23:14:12.747879 25413 raft_consensus.cc:2229] T 16c60d1376eb4a2dbb7325a1633f94cb P f9080c60e3e440999f4ed4fa4e342d8a [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:14:12.748564 25413 raft_consensus.cc:2258] T 16c60d1376eb4a2dbb7325a1633f94cb P f9080c60e3e440999f4ed4fa4e342d8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:14:12.751869 25387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:12.728105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41546 (local address 127.10.191.193:34215)
0620 23:14:12.728292 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:12.728303 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:12.731480 (+ 3177us) server_negotiation.cc:407] Connection header received
0620 23:14:12.731763 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:12.731769 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:12.731847 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:12.731936 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:12.734159 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:12.735341 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:12.737513 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:12.738207 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:12.738327 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:12.744789 (+ 6462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:12.744804 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:12.744821 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:12.744863 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:12.748778 (+ 3915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:12.749562 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:12.749570 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:12.749574 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:12.749719 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:12.750057 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:12.750065 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:12.750071 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:12.750534 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:12.750854 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:12.751259 (+ 405us) server_negotiation.cc:299] Negotiation successful
0620 23:14:12.751486 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 23:14:12.752600 25098 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:14:12.752884 25413 ts_tablet_manager.cc:1905] T 16c60d1376eb4a2dbb7325a1633f94cb P f9080c60e3e440999f4ed4fa4e342d8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:14:12.760689 25413 ts_tablet_manager.cc:1918] T 16c60d1376eb4a2dbb7325a1633f94cb P f9080c60e3e440999f4ed4fa4e342d8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:14:12.760866 25413 log.cc:1196] T 16c60d1376eb4a2dbb7325a1633f94cb P f9080c60e3e440999f4ed4fa4e342d8a: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/wal/wals/16c60d1376eb4a2dbb7325a1633f94cb
I20260620 23:14:12.761447 25413 ts_tablet_manager.cc:1939] T 16c60d1376eb4a2dbb7325a1633f94cb P f9080c60e3e440999f4ed4fa4e342d8a: Deleting consensus metadata
I20260620 23:14:12.763298 24899 catalog_manager.cc:4841] TS f9080c60e3e440999f4ed4fa4e342d8a (127.10.191.194:46539): tablet 16c60d1376eb4a2dbb7325a1633f94cb (table test-table [id=8fd887122364412ba23579790beeb581]) successfully deleted
I20260620 23:14:12.841960 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 24975
I20260620 23:14:12.859882 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 25109
Jun 20 23:14:12 dist-test-slave-jkbq krb5kdc[24869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.194@KRBTEST.COM
I20260620 23:14:12.876622 25383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:12.860684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47968 (local address 127.10.191.194:46539)
0620 23:14:12.860887 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:12.860893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:12.860966 (+ 73us) server_negotiation.cc:407] Connection header received
0620 23:14:12.861127 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:12.861133 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:12.861223 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:12.861334 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:12.862559 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:12.863806 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:12.865096 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:12.865785 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:12.865916 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:12.870450 (+ 4534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:12.870462 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:12.870476 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:12.870516 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:12.873626 (+ 3110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:12.874404 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:12.874411 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:12.874413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:12.874553 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:12.875129 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:12.875133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:12.875135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:12.875483 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:12.875810 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:12.876107 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 23:14:12.876286 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:14:12.877525 25232 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:14:12.988932 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 25109
I20260620 23:14:13.007221 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 25242
Jun 20 23:14:13 dist-test-slave-jkbq krb5kdc[24869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.195@KRBTEST.COM
I20260620 23:14:13.023334 25385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:13.008134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56552 (local address 127.10.191.195:46283)
0620 23:14:13.008343 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:13.008349 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:13.008463 (+ 114us) server_negotiation.cc:407] Connection header received
0620 23:14:13.008696 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:13.008700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:13.008764 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:13.008893 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:13.010190 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:13.011054 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:13.012436 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:13.012840 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:13.012974 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:13.017397 (+ 4423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:13.017412 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:13.017428 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:13.017462 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:13.020441 (+ 2979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:13.021157 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:13.021165 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:13.021167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:13.021336 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:13.021722 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:13.021728 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:13.021731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:13.022096 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:13.022438 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:13.022717 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 23:14:13.023009 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 23:14:13.024041 25365 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:14:13.114528 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 25242
I20260620 23:14:13.131538 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 24882
I20260620 23:14:13.132730 24944 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:14:13.243364 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 24882
I20260620 23:14:13.264194 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:43585
--webserver_interface=127.10.191.254
--webserver_port=39291
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:39607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:43585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:14:13.582525 25419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:14:13.582857 25419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:14:13.583045 25419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:14:13.590021 25419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:14:13.590161 25419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:14:13.590277 25419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:14:13.590381 25419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:14:13.590473 25419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:14:13.599906 25419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:39607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:43585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:43585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=39291
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:14:13.601128 25419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:14:13.602396 25419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:14:13.624413 25426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:13.624439 25425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:13.624486 25428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:14:13.625286 25419 server_base.cc:1034] running on GCE node
I20260620 23:14:13.626160 25419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:14:13.627748 25419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:14:13.628993 25419 hybrid_clock.cc:648] HybridClock initialized: now 1781997253628938 us; error 64 us; skew 500 ppm
I20260620 23:14:13.629403 25419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:14:13 dist-test-slave-jkbq krb5kdc[24869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997253, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:14:13.632942 25419 init.cc:377] Logged in from keytab as kudu/127.10.191.254@KRBTEST.COM (short username kudu)
I20260620 23:14:13.634606 25419 webserver.cc:460] Webserver started at http://127.10.191.254:39291/ using document root <none> and password file <none>
I20260620 23:14:13.635442 25419 fs_manager.cc:359] Metadata directory not provided
I20260620 23:14:13.635673 25419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:14:13.641947 25419 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260620 23:14:13.645025 25435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:13.646445 25419 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:14:13.646629 25419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "5cc88259af6a46a7be7375becd6aef0f"
format_stamp: "Formatted at 2026-06-20 23:14:10 on dist-test-slave-jkbq"
server_key: "f23fe6202a7f8300f94db4d3a4b13825"
server_key_iv: "61f9b18d91e0633c9112f3abd4516173"
server_key_version: "encryptionkey@0"
I20260620 23:14:13.647750 25419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:14:13.662134 25419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:14:13.666582 25419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:14:13.666894 25419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:14:13.685683 25419 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:43585
I20260620 23:14:13.685710 25486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:43585 every 8 connection(s)
I20260620 23:14:13.687211 25419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:14:13.694000 25487 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:14:13.695232 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 25419
I20260620 23:14:13.696010 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.193:34215
--local_ip_for_outbound_sockets=127.10.191.193
--tserver_master_addrs=127.10.191.254:43585
--webserver_port=37433
--webserver_interface=127.10.191.193
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:39607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:14:13.697476 25487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f: Bootstrap starting.
I20260620 23:14:13.716857 25487 log.cc:824] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f: Log is configured to *not* fsync() on all Append() calls
I20260620 23:14:13.729050 25487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:14:13.729594 25487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f: Bootstrap complete.
I20260620 23:14:13.738919 25487 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc88259af6a46a7be7375becd6aef0f" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43585 } }
I20260620 23:14:13.739810 25487 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cc88259af6a46a7be7375becd6aef0f, State: Initialized, Role: FOLLOWER
I20260620 23:14:13.740463 25487 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc88259af6a46a7be7375becd6aef0f" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43585 } }
I20260620 23:14:13.740715 25487 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:14:13.740855 25487 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:14:13.741039 25487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:14:13.744877 25487 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc88259af6a46a7be7375becd6aef0f" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43585 } }
I20260620 23:14:13.745402 25487 leader_election.cc:302] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cc88259af6a46a7be7375becd6aef0f; no voters:
I20260620 23:14:13.746011 25487 leader_election.cc:288] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 23:14:13.746212 25491 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [term 2 FOLLOWER]: Leader election won for term 2
I20260620 23:14:13.747493 25491 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [term 2 LEADER]: Becoming Leader. State: Replica: 5cc88259af6a46a7be7375becd6aef0f, State: Running, Role: LEADER
I20260620 23:14:13.748188 25487 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:14:13.748044 25491 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc88259af6a46a7be7375becd6aef0f" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43585 } }
I20260620 23:14:13.751281 25493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cc88259af6a46a7be7375becd6aef0f. Latest consensus state: current_term: 2 leader_uuid: "5cc88259af6a46a7be7375becd6aef0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc88259af6a46a7be7375becd6aef0f" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43585 } } }
I20260620 23:14:13.751641 25493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [sys.catalog]: This master's current role is: LEADER
I20260620 23:14:13.752079 25492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5cc88259af6a46a7be7375becd6aef0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc88259af6a46a7be7375becd6aef0f" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43585 } } }
I20260620 23:14:13.752339 25492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [sys.catalog]: This master's current role is: LEADER
I20260620 23:14:13.760249 25499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:14:13.768288 25499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:14:13.769758 25499 catalog_manager.cc:1259] Loaded cluster ID: b8624621d19c4ecc95b33ee0bd1970fb
I20260620 23:14:13.769873 25499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:14:13.772822 25499 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 23:14:13.788138 25509 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 23:14:13.790750 25499 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 23:14:13.799829 25499 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:14:13.801545 25499 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f: Loaded TSK: 0
I20260620 23:14:13.803169 25499 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 23:14:14.034303 25489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:14:14.034612 25489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:14:14.034770 25489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:14:14.041846 25489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:14:14.041991 25489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:14:14.042155 25489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.193
I20260620 23:14:14.052011 25489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:39607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.193:34215
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.191.193
--webserver_port=37433
--tserver_master_addrs=127.10.191.254:43585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.193
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:14:14.053260 25489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:14:14.054541 25489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:14:14.070765 25516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:14.070811 25515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:14.071869 25518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:14:14.076877 25489 server_base.cc:1034] running on GCE node
I20260620 23:14:14.077615 25489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:14:14.078651 25489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:14:14.079887 25489 hybrid_clock.cc:648] HybridClock initialized: now 1781997254079810 us; error 73 us; skew 500 ppm
I20260620 23:14:14.080283 25489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:14:14 dist-test-slave-jkbq krb5kdc[24869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997254, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:14:14.083706 25489 init.cc:377] Logged in from keytab as kudu/127.10.191.193@KRBTEST.COM (short username kudu)
I20260620 23:14:14.085409 25489 webserver.cc:460] Webserver started at http://127.10.191.193:37433/ using document root <none> and password file <none>
I20260620 23:14:14.086148 25489 fs_manager.cc:359] Metadata directory not provided
I20260620 23:14:14.086306 25489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:14:14.091936 25489 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 23:14:14.094887 25525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:14.096374 25489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:14:14.096570 25489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/wal
uuid: "2449fe0928b543e49e272470c621a01a"
format_stamp: "Formatted at 2026-06-20 23:14:11 on dist-test-slave-jkbq"
server_key: "0f6302b1228b49492a3c285bcecbdfc4"
server_key_iv: "310538ec03f0d7d0ccfa68a53f874241"
server_key_version: "encryptionkey@0"
I20260620 23:14:14.097577 25489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:14:14.116127 25489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:14:14.120723 25489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:14:14.121017 25489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:14:14.122124 25489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:14:14.123646 25489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:14:14.123761 25489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:14.123889 25489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:14:14.123977 25489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:14.158459 25489 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.193:34215
I20260620 23:14:14.158517 25637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.193:34215 every 8 connection(s)
I20260620 23:14:14.160025 25489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
I20260620 23:14:14.168483 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 25489
I20260620 23:14:14.169268 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.194:46539
--local_ip_for_outbound_sockets=127.10.191.194
--tserver_master_addrs=127.10.191.254:43585
--webserver_port=35681
--webserver_interface=127.10.191.194
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:39607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:14:14 dist-test-slave-jkbq krb5kdc[24869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997254, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:14:14.184996 25640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:14.163126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:45641 (local address 127.10.191.254:43585)
0620 23:14:14.163668 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:14.163680 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:14.164464 (+ 784us) server_negotiation.cc:407] Connection header received
0620 23:14:14.166546 (+ 2082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:14.166560 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:14.166991 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:14.167296 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:14.168747 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:14.170809 (+ 2062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:14.172368 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:14.173499 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:14.173694 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:14.177441 (+ 3747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:14.177448 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:14.177461 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:14.177496 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:14.180425 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:14.181376 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:14.181383 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:14.181389 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:14.181549 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:14.182098 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:14.182102 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:14.182104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:14.182585 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:14.182727 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:14.183727 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0620 23:14:14.184056 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":231,"threads_started":1}
I20260620 23:14:14.185395 25641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:14.163377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.193:45641)
0620 23:14:14.164256 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:14.164308 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:14.166030 (+ 1722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:14.167444 (+ 1414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:14.167451 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:14.167945 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:14:14.168482 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:14.168500 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:14.170992 (+ 2492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:14.170996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:14.172202 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:14.172214 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:14.173706 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:14.173711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:14.173906 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:14.174599 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:14:14.174620 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:14:14.177160 (+ 2540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:14:14.180570 (+ 3410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:14.180582 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:14.180600 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:14.181208 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:14.181685 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:14.181694 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:14.181701 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:14.181989 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:14.182738 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:14:14.182748 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:14:14.183217 (+ 469us) client_negotiation.cc:769] Sending connection context
0620 23:14:14.183520 (+ 303us) client_negotiation.cc:240] Negotiation successful
0620 23:14:14.184073 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":353,"threads_started":1}
I20260620 23:14:14.188409 25638 heartbeater.cc:347] Connected to a master server at 127.10.191.254:43585
I20260620 23:14:14.188717 25638 heartbeater.cc:464] Registering TS with master...
I20260620 23:14:14.189471 25638 heartbeater.cc:511] Master 127.10.191.254:43585 requested a full tablet report, sending...
I20260620 23:14:14.191658 25452 ts_manager.cc:194] Registered new tserver with Master: 2449fe0928b543e49e272470c621a01a (127.10.191.193:34215)
I20260620 23:14:14.193143 25452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.193@KRBTEST.COM'} at 127.10.191.193:45641
W20260620 23:14:14.495133 25642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:14:14.495437 25642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:14:14.495609 25642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:14:14.502612 25642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:14:14.502743 25642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:14:14.502936 25642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.194
I20260620 23:14:14.512210 25642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:39607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.194:46539
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.191.194
--webserver_port=35681
--tserver_master_addrs=127.10.191.254:43585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.194
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:14:14.513453 25642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:14:14.514742 25642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:14:14.527621 25648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:14.527846 25647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:14.527807 25650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:14:14.528740 25642 server_base.cc:1034] running on GCE node
I20260620 23:14:14.529306 25642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:14:14.530191 25642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:14:14.531424 25642 hybrid_clock.cc:648] HybridClock initialized: now 1781997254531351 us; error 50 us; skew 500 ppm
I20260620 23:14:14.531721 25642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:14:14 dist-test-slave-jkbq krb5kdc[24869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997254, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:14:14.534634 25642 init.cc:377] Logged in from keytab as kudu/127.10.191.194@KRBTEST.COM (short username kudu)
I20260620 23:14:14.536139 25642 webserver.cc:460] Webserver started at http://127.10.191.194:35681/ using document root <none> and password file <none>
I20260620 23:14:14.536785 25642 fs_manager.cc:359] Metadata directory not provided
I20260620 23:14:14.536938 25642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:14:14.541926 25642 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 23:14:14.544549 25657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:14.545787 25642 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:14:14.545964 25642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/wal
uuid: "f9080c60e3e440999f4ed4fa4e342d8a"
format_stamp: "Formatted at 2026-06-20 23:14:11 on dist-test-slave-jkbq"
server_key: "f18e0ba4d25707f3a5b5713a61a69a62"
server_key_iv: "36314dcd557976ff2644feb0dc00c42d"
server_key_version: "encryptionkey@0"
I20260620 23:14:14.546918 25642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:14:14.571018 25642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:14:14.574977 25642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:14:14.575243 25642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:14:14.576185 25642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:14:14.577533 25642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:14:14.577651 25642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:14.577780 25642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:14:14.577863 25642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:14.610747 25642 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.194:46539
I20260620 23:14:14.610831 25769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.194:46539 every 8 connection(s)
I20260620 23:14:14.612193 25642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
I20260620 23:14:14.613394 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 25642
I20260620 23:14:14.614101 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.195:46283
--local_ip_for_outbound_sockets=127.10.191.195
--tserver_master_addrs=127.10.191.254:43585
--webserver_port=41539
--webserver_interface=127.10.191.195
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:39607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:14:14 dist-test-slave-jkbq krb5kdc[24869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997254, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:14:14.633952 25640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:14.615006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:46093 (local address 127.10.191.254:43585)
0620 23:14:14.616977 (+ 1971us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:14.616985 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:14.617013 (+ 28us) server_negotiation.cc:407] Connection header received
0620 23:14:14.618329 (+ 1316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:14.618338 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:14.618439 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:14.618558 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:14.620336 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:14.621030 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:14.622574 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:14.623080 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:14.623218 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:14.626815 (+ 3597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:14.626825 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:14.626833 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:14.626899 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:14.630462 (+ 3563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:14.631262 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:14.631269 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:14.631272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:14.631436 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:14.631896 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:14.631904 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:14.631910 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:14.632325 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:14.632501 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:14.633186 (+ 685us) server_negotiation.cc:299] Negotiation successful
0620 23:14:14.633465 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1805}
I20260620 23:14:14.634488 25772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:14.615576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.194:46093)
0620 23:14:14.616373 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:14.616416 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:14.618044 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:14.618764 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:14.618772 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:14.619351 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:14:14.620146 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:14.620166 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:14.621186 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:14.621191 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:14.622419 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:14.622429 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:14.623306 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:14.623313 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:14.623564 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:14.624280 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:14:14.624305 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:14:14.626589 (+ 2284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:14:14.630603 (+ 4014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:14.630609 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:14.630620 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:14.631140 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:14.631565 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:14.631570 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:14.631574 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:14.631758 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:14.632477 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:14:14.632483 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:14:14.632783 (+ 300us) client_negotiation.cc:769] Sending connection context
0620 23:14:14.633011 (+ 228us) client_negotiation.cc:240] Negotiation successful
0620 23:14:14.633387 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":334,"threads_started":1}
I20260620 23:14:14.636164 25770 heartbeater.cc:347] Connected to a master server at 127.10.191.254:43585
I20260620 23:14:14.636466 25770 heartbeater.cc:464] Registering TS with master...
I20260620 23:14:14.637164 25770 heartbeater.cc:511] Master 127.10.191.254:43585 requested a full tablet report, sending...
I20260620 23:14:14.638622 25452 ts_manager.cc:194] Registered new tserver with Master: f9080c60e3e440999f4ed4fa4e342d8a (127.10.191.194:46539)
I20260620 23:14:14.639528 25452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.194@KRBTEST.COM'} at 127.10.191.194:46093
W20260620 23:14:14.938805 25773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:14:14.939133 25773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:14:14.939325 25773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:14:14.946455 25773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:14:14.946616 25773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:14:14.946808 25773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.195
I20260620 23:14:14.956830 25773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:39607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.195:46283
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.191.195
--webserver_port=41539
--tserver_master_addrs=127.10.191.254:43585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.195
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:14:14.958048 25773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:14:14.959417 25773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:14:14.972694 25778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:14.972772 25779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:14.972744 25781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:14:14.974206 25773 server_base.cc:1034] running on GCE node
I20260620 23:14:14.974802 25773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:14:14.975744 25773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:14:14.976967 25773 hybrid_clock.cc:648] HybridClock initialized: now 1781997254976907 us; error 54 us; skew 500 ppm
I20260620 23:14:14.977286 25773 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:14:14 dist-test-slave-jkbq krb5kdc[24869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997254, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:14:14.980307 25773 init.cc:377] Logged in from keytab as kudu/127.10.191.195@KRBTEST.COM (short username kudu)
I20260620 23:14:14.981669 25773 webserver.cc:460] Webserver started at http://127.10.191.195:41539/ using document root <none> and password file <none>
I20260620 23:14:14.982344 25773 fs_manager.cc:359] Metadata directory not provided
I20260620 23:14:14.982496 25773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:14:14.987986 25773 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 23:14:14.990495 25788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:14.991786 25773 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:14:14.991962 25773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/wal
uuid: "444f05061ce9490cb5725fc402922b2d"
format_stamp: "Formatted at 2026-06-20 23:14:12 on dist-test-slave-jkbq"
server_key: "08ea9c8bc3c44b03c0bc58960f40e04e"
server_key_iv: "3efdd9e69e74f58bfdffe17f4820d804"
server_key_version: "encryptionkey@0"
I20260620 23:14:14.992890 25773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:14:15.011693 25773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:14:15.015420 25773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:14:15.015686 25773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:14:15.016618 25773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:14:15.017906 25773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:14:15.018025 25773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:15.018157 25773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:14:15.018244 25773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:15.051378 25773 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.195:46283
I20260620 23:14:15.051542 25900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.195:46283 every 8 connection(s)
I20260620 23:14:15.052876 25773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
I20260620 23:14:15.058526 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 25773
Jun 20 23:14:15 dist-test-slave-jkbq krb5kdc[24869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997254, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:14:15.075978 25905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:15.059444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48390 (local address 127.10.191.254:43585)
0620 23:14:15.060547 (+ 1103us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:15.060553 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:15.060573 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:14:15.060627 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:15.060632 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:15.060705 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:15.060806 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:15.063203 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:15.064003 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:15.065592 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:15.066101 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:15.066198 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:15.068908 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:15.068914 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:15.068919 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:15.068965 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:15.072588 (+ 3623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:15.073446 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:15.073455 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:15.073458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:15.073657 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:15.074247 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:15.074253 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:15.074256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:15.074777 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:15.075044 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:15.075400 (+ 356us) server_negotiation.cc:299] Negotiation successful
0620 23:14:15.075575 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":418,"server-negotiator.queue_time_us":984,"spinlock_wait_cycles":6400,"thread_start_us":1658,"threads_started":1}
I20260620 23:14:15.076484 25640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:15.055467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:54757 (local address 127.10.191.254:43585)
0620 23:14:15.055650 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:15.055654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:15.056763 (+ 1109us) server_negotiation.cc:407] Connection header received
0620 23:14:15.058550 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:15.058556 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:15.058651 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:15.058773 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:15.060740 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:15.062437 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:15.064286 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:15.064697 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:15.064832 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:15.068535 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:15.068541 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:15.068552 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:15.068581 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:15.072520 (+ 3939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:15.073469 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:15.073472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:15.073475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:15.073657 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:15.074257 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:15.074261 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:15.074264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:15.074777 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:15.074954 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:15.075883 (+ 929us) server_negotiation.cc:299] Negotiation successful
0620 23:14:15.076103 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":64,"spinlock_wait_cycles":23552}
I20260620 23:14:15.077634 25903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:15.055931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.195:54757)
0620 23:14:15.056605 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:15.056643 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:15.058258 (+ 1615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:15.059043 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:15.059051 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:15.059653 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:14:15.060496 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:15.060521 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:15.062641 (+ 2120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:15.062647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:15.064137 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:15.064152 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:15.064931 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:15.064938 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:15.065192 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:15.065850 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:14:15.065877 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:14:15.068332 (+ 2455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:14:15.072759 (+ 4427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:15.072768 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:15.072786 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:15.073358 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:15.073808 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:15.073813 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:15.073817 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:15.074126 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:15.074949 (+ 823us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:14:15.074957 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:14:15.075426 (+ 469us) client_negotiation.cc:769] Sending connection context
0620 23:14:15.075762 (+ 336us) client_negotiation.cc:240] Negotiation successful
0620 23:14:15.076327 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":253,"threads_started":1}
I20260620 23:14:15.079082 25901 heartbeater.cc:347] Connected to a master server at 127.10.191.254:43585
I20260620 23:14:15.079394 25901 heartbeater.cc:464] Registering TS with master...
I20260620 23:14:15.080102 25901 heartbeater.cc:511] Master 127.10.191.254:43585 requested a full tablet report, sending...
I20260620 23:14:15.081526 25451 ts_manager.cc:194] Registered new tserver with Master: 444f05061ce9490cb5725fc402922b2d (127.10.191.195:46283)
I20260620 23:14:15.082444 25451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.195@KRBTEST.COM'} at 127.10.191.195:54757
I20260620 23:14:15.090780 11007 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:14:15.098063 25640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:15.091702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48400 (local address 127.10.191.254:43585)
0620 23:14:15.091883 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:15.091887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:15.092459 (+ 572us) server_negotiation.cc:407] Connection header received
0620 23:14:15.092613 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:15.092617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:15.092766 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:15.092882 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:14:15.093292 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:15.094026 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:15.095585 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:15.096100 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:15.096214 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:15.096894 (+ 680us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:14:15.097520 (+ 626us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:14:15.097635 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:15.097798 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 23:14:15.097867 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 23:14:15.099031 25451 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:48400:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:14:15.106408 25451 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:14:15.130952 25911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:15.118350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.194:46539 (local address 127.0.0.1:44824)
0620 23:14:15.119422 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:15.119471 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:15.119653 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:15.120711 (+ 1058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:15.120721 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:15.120764 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:14:15.121324 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:15.121341 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:15.123973 (+ 2632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:15.123981 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:15.127265 (+ 3284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:15.127290 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:15.129819 (+ 2529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:15.129825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:15.130043 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:15.130105 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 23:14:15.130227 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 23:14:15.130435 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"thread_start_us":559,"threads_started":1}
I20260620 23:14:15.131394 25912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:15.118705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44824 (local address 127.10.191.194:46539)
0620 23:14:15.119734 (+ 1029us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:15.119744 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:15.119779 (+ 35us) server_negotiation.cc:407] Connection header received
0620 23:14:15.119881 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:15.119893 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:15.120187 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:15.120371 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:14:15.121625 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:15.123716 (+ 2091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:15.127525 (+ 3809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:15.129586 (+ 2061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:15.129788 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:15.129994 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:15.130599 (+ 605us) server_negotiation.cc:299] Negotiation successful
0620 23:14:15.130789 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":842,"thread_start_us":708,"threads_started":1}
I20260620 23:14:15.135676 25705 tablet_service.cc:1467] Processing CreateTablet for tablet 1330d571bc65478bb6fca56c2fe12d47 (DEFAULT_TABLE table=test-table [id=7f3c0e5e57764062903ffa41092a5381]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:14:15.137130 25705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1330d571bc65478bb6fca56c2fe12d47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:14:15.137187 25913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:15.119790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.195:46283 (local address 127.0.0.1:58916)
0620 23:14:15.120372 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:15.120391 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:15.120514 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:15.130491 (+ 9977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:15.130496 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:15.130527 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:14:15.130967 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:15.130981 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:15.133142 (+ 2161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:15.133147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:15.134501 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:15.134516 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:15.136374 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:15.136380 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:15.136567 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:15.136600 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 23:14:15.136706 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 23:14:15.136843 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":312,"threads_started":1}
I20260620 23:14:15.137512 25914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:15.120189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58916 (local address 127.10.191.195:46283)
0620 23:14:15.129813 (+ 9624us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:15.129823 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:15.129863 (+ 40us) server_negotiation.cc:407] Connection header received
0620 23:14:15.129964 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:15.129973 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:15.130267 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:15.131269 (+ 1002us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:14:15.131570 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:15.133007 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:15.134704 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:15.136133 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:15.136406 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:15.136593 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:15.136764 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 23:14:15.136932 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9442,"thread_start_us":9272,"threads_started":1}
I20260620 23:14:15.139190 25915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:15.120395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:34215 (local address 127.0.0.1:50820)
0620 23:14:15.129364 (+ 8969us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:15.129387 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:15.130115 (+ 728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:15.130820 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:15.130825 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:15.130850 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:14:15.131430 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:15.131445 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:15.134398 (+ 2953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:15.134404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:15.136493 (+ 2089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:15.136506 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:15.138397 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:15.138403 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:15.138601 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:15.138633 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 23:14:15.138723 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 23:14:15.138843 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8846,"thread_start_us":8708,"threads_started":1}
I20260620 23:14:15.139328 25916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:15.120891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50820 (local address 127.10.191.193:34215)
0620 23:14:15.129068 (+ 8177us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:15.129079 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:15.130257 (+ 1178us) server_negotiation.cc:407] Connection header received
0620 23:14:15.130375 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:15.130382 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:15.130604 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:15.131446 (+ 842us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:14:15.132721 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:15.134237 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:15.136704 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:15.138171 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:15.138341 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:15.138514 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:15.138818 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 23:14:15.139003 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7997,"thread_start_us":7856,"threads_started":1}
I20260620 23:14:15.141229 25836 tablet_service.cc:1467] Processing CreateTablet for tablet 1330d571bc65478bb6fca56c2fe12d47 (DEFAULT_TABLE table=test-table [id=7f3c0e5e57764062903ffa41092a5381]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:14:15.142611 25836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1330d571bc65478bb6fca56c2fe12d47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:14:15.142741 25573 tablet_service.cc:1467] Processing CreateTablet for tablet 1330d571bc65478bb6fca56c2fe12d47 (DEFAULT_TABLE table=test-table [id=7f3c0e5e57764062903ffa41092a5381]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:14:15.144006 25573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1330d571bc65478bb6fca56c2fe12d47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:14:15.149614 25917 tablet_bootstrap.cc:492] T 1330d571bc65478bb6fca56c2fe12d47 P f9080c60e3e440999f4ed4fa4e342d8a: Bootstrap starting.
I20260620 23:14:15.152949 25917 tablet_bootstrap.cc:654] T 1330d571bc65478bb6fca56c2fe12d47 P f9080c60e3e440999f4ed4fa4e342d8a: Neither blocks nor log segments found. Creating new log.
I20260620 23:14:15.153957 25917 log.cc:824] T 1330d571bc65478bb6fca56c2fe12d47 P f9080c60e3e440999f4ed4fa4e342d8a: Log is configured to *not* fsync() on all Append() calls
I20260620 23:14:15.157284 25917 tablet_bootstrap.cc:492] T 1330d571bc65478bb6fca56c2fe12d47 P f9080c60e3e440999f4ed4fa4e342d8a: No bootstrap required, opened a new log
I20260620 23:14:15.157590 25917 ts_tablet_manager.cc:1397] T 1330d571bc65478bb6fca56c2fe12d47 P f9080c60e3e440999f4ed4fa4e342d8a: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260620 23:14:15.158128 25919 tablet_bootstrap.cc:492] T 1330d571bc65478bb6fca56c2fe12d47 P 444f05061ce9490cb5725fc402922b2d: Bootstrap starting.
I20260620 23:14:15.161957 25919 tablet_bootstrap.cc:654] T 1330d571bc65478bb6fca56c2fe12d47 P 444f05061ce9490cb5725fc402922b2d: Neither blocks nor log segments found. Creating new log.
I20260620 23:14:15.163427 25919 log.cc:824] T 1330d571bc65478bb6fca56c2fe12d47 P 444f05061ce9490cb5725fc402922b2d: Log is configured to *not* fsync() on all Append() calls
I20260620 23:14:15.171707 25920 tablet_bootstrap.cc:492] T 1330d571bc65478bb6fca56c2fe12d47 P 2449fe0928b543e49e272470c621a01a: Bootstrap starting.
I20260620 23:14:15.170401 25917 raft_consensus.cc:348] T 1330d571bc65478bb6fca56c2fe12d47 P f9080c60e3e440999f4ed4fa4e342d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9080c60e3e440999f4ed4fa4e342d8a" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 46539 } } peers { permanent_uuid: "444f05061ce9490cb5725fc402922b2d" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46283 } } peers { permanent_uuid: "2449fe0928b543e49e272470c621a01a" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 34215 } }
I20260620 23:14:15.172576 25917 raft_consensus.cc:374] T 1330d571bc65478bb6fca56c2fe12d47 P f9080c60e3e440999f4ed4fa4e342d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:14:15.172698 25917 raft_consensus.cc:729] T 1330d571bc65478bb6fca56c2fe12d47 P f9080c60e3e440999f4ed4fa4e342d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9080c60e3e440999f4ed4fa4e342d8a, State: Initialized, Role: FOLLOWER
I20260620 23:14:15.173319 25917 consensus_queue.cc:260] T 1330d571bc65478bb6fca56c2fe12d47 P f9080c60e3e440999f4ed4fa4e342d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9080c60e3e440999f4ed4fa4e342d8a" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 46539 } } peers { permanent_uuid: "444f05061ce9490cb5725fc402922b2d" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46283 } } peers { permanent_uuid: "2449fe0928b543e49e272470c621a01a" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 34215 } }
I20260620 23:14:15.174520 25919 tablet_bootstrap.cc:492] T 1330d571bc65478bb6fca56c2fe12d47 P 444f05061ce9490cb5725fc402922b2d: No bootstrap required, opened a new log
I20260620 23:14:15.174896 25919 ts_tablet_manager.cc:1397] T 1330d571bc65478bb6fca56c2fe12d47 P 444f05061ce9490cb5725fc402922b2d: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.009s
I20260620 23:14:15.176359 25920 tablet_bootstrap.cc:654] T 1330d571bc65478bb6fca56c2fe12d47 P 2449fe0928b543e49e272470c621a01a: Neither blocks nor log segments found. Creating new log.
I20260620 23:14:15.177456 25920 log.cc:824] T 1330d571bc65478bb6fca56c2fe12d47 P 2449fe0928b543e49e272470c621a01a: Log is configured to *not* fsync() on all Append() calls
I20260620 23:14:15.181674 25917 ts_tablet_manager.cc:1428] T 1330d571bc65478bb6fca56c2fe12d47 P f9080c60e3e440999f4ed4fa4e342d8a: Time spent starting tablet: real 0.024s user 0.010s sys 0.010s
I20260620 23:14:15.182159 25770 heartbeater.cc:503] Master 127.10.191.254:43585 was elected leader, sending a full tablet report...
I20260620 23:14:15.184486 25920 tablet_bootstrap.cc:492] T 1330d571bc65478bb6fca56c2fe12d47 P 2449fe0928b543e49e272470c621a01a: No bootstrap required, opened a new log
I20260620 23:14:15.184758 25920 ts_tablet_manager.cc:1397] T 1330d571bc65478bb6fca56c2fe12d47 P 2449fe0928b543e49e272470c621a01a: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260620 23:14:15.186470 25919 raft_consensus.cc:348] T 1330d571bc65478bb6fca56c2fe12d47 P 444f05061ce9490cb5725fc402922b2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9080c60e3e440999f4ed4fa4e342d8a" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 46539 } } peers { permanent_uuid: "444f05061ce9490cb5725fc402922b2d" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46283 } } peers { permanent_uuid: "2449fe0928b543e49e272470c621a01a" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 34215 } }
I20260620 23:14:15.186833 25919 raft_consensus.cc:374] T 1330d571bc65478bb6fca56c2fe12d47 P 444f05061ce9490cb5725fc402922b2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:14:15.186949 25919 raft_consensus.cc:729] T 1330d571bc65478bb6fca56c2fe12d47 P 444f05061ce9490cb5725fc402922b2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 444f05061ce9490cb5725fc402922b2d, State: Initialized, Role: FOLLOWER
I20260620 23:14:15.187568 25919 consensus_queue.cc:260] T 1330d571bc65478bb6fca56c2fe12d47 P 444f05061ce9490cb5725fc402922b2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9080c60e3e440999f4ed4fa4e342d8a" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 46539 } } peers { permanent_uuid: "444f05061ce9490cb5725fc402922b2d" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46283 } } peers { permanent_uuid: "2449fe0928b543e49e272470c621a01a" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 34215 } }
I20260620 23:14:15.188722 25901 heartbeater.cc:503] Master 127.10.191.254:43585 was elected leader, sending a full tablet report...
I20260620 23:14:15.189203 25919 ts_tablet_manager.cc:1428] T 1330d571bc65478bb6fca56c2fe12d47 P 444f05061ce9490cb5725fc402922b2d: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260620 23:14:15.193699 25920 raft_consensus.cc:348] T 1330d571bc65478bb6fca56c2fe12d47 P 2449fe0928b543e49e272470c621a01a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9080c60e3e440999f4ed4fa4e342d8a" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 46539 } } peers { permanent_uuid: "444f05061ce9490cb5725fc402922b2d" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46283 } } peers { permanent_uuid: "2449fe0928b543e49e272470c621a01a" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 34215 } }
I20260620 23:14:15.194025 25920 raft_consensus.cc:374] T 1330d571bc65478bb6fca56c2fe12d47 P 2449fe0928b543e49e272470c621a01a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:14:15.194094 25920 raft_consensus.cc:729] T 1330d571bc65478bb6fca56c2fe12d47 P 2449fe0928b543e49e272470c621a01a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2449fe0928b543e49e272470c621a01a, State: Initialized, Role: FOLLOWER
I20260620 23:14:15.194563 25920 consensus_queue.cc:260] T 1330d571bc65478bb6fca56c2fe12d47 P 2449fe0928b543e49e272470c621a01a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9080c60e3e440999f4ed4fa4e342d8a" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 46539 } } peers { permanent_uuid: "444f05061ce9490cb5725fc402922b2d" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46283 } } peers { permanent_uuid: "2449fe0928b543e49e272470c621a01a" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 34215 } }
I20260620 23:14:15.195489 25638 heartbeater.cc:503] Master 127.10.191.254:43585 was elected leader, sending a full tablet report...
I20260620 23:14:15.196074 25920 ts_tablet_manager.cc:1428] T 1330d571bc65478bb6fca56c2fe12d47 P 2449fe0928b543e49e272470c621a01a: Time spent starting tablet: real 0.011s user 0.006s sys 0.006s
I20260620 23:14:15.215168 25925 raft_consensus.cc:482] T 1330d571bc65478bb6fca56c2fe12d47 P 2449fe0928b543e49e272470c621a01a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:14:15.215477 25925 raft_consensus.cc:504] T 1330d571bc65478bb6fca56c2fe12d47 P 2449fe0928b543e49e272470c621a01a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9080c60e3e440999f4ed4fa4e342d8a" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 46539 } } peers { permanent_uuid: "444f05061ce9490cb5725fc402922b2d" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46283 } } peers { permanent_uuid: "2449fe0928b543e49e272470c621a01a" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 34215 } }
I20260620 23:14:15.217298 25925 leader_election.cc:288] T 1330d571bc65478bb6fca56c2fe12d47 P 2449fe0928b543e49e272470c621a01a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9080c60e3e440999f4ed4fa4e342d8a (127.10.191.194:46539), 444f05061ce9490cb5725fc402922b2d (127.10.191.195:46283)
I20260620 23:14:15.224522 25912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:15.217744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:41695 (local address 127.10.191.194:46539)
0620 23:14:15.217934 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:15.217939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:15.218463 (+ 524us) server_negotiation.cc:407] Connection header received
0620 23:14:15.218678 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:15.218686 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:15.218776 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:15.218896 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:14:15.219571 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:15.220470 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:15.222466 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:15.223562 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:15.223685 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:15.223738 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:15.224129 (+ 391us) server_negotiation.cc:299] Negotiation successful
0620 23:14:15.224256 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 23:14:15.224718 25926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:15.217776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.194:46539 (local address 127.10.191.193:41695)
0620 23:14:15.218353 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:15.218369 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:15.218537 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:15.218982 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:15.218986 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:15.219009 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:14:15.219426 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:15.219438 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:15.220605 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:15.220610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:15.222305 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:15.222317 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:15.223761 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:15.223770 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:15.223961 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:15.223999 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 23:14:15.224100 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 23:14:15.224241 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":24,"thread_start_us":296,"threads_started":1}
I20260620 23:14:15.225193 25914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:15.218608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:44431 (local address 127.10.191.195:46283)
0620 23:14:15.218849 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:15.218855 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:15.219214 (+ 359us) server_negotiation.cc:407] Connection header received
0620 23:14:15.219325 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:15.219338 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:15.219427 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:15.219532 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:14:15.220088 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:15.221002 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:15.223064 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:15.224138 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:15.224275 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:15.224338 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:15.224818 (+ 480us) server_negotiation.cc:299] Negotiation successful
0620 23:14:15.224940 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 23:14:15.225281 25927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:15.218485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.195:46283 (local address 127.10.191.193:44431)
0620 23:14:15.219107 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:15.219120 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:15.219220 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:15.219560 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:15.219564 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:15.219588 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:14:15.219925 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:15.219936 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:15.221141 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:15.221144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:15.222908 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:15.222923 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:15.224392 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:15.224400 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:15.224646 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:15.224690 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 23:14:15.224787 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 23:14:15.224905 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"mutex_wait_us":1,"thread_start_us":279,"threads_started":1}
I20260620 23:14:15.225811 25725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1330d571bc65478bb6fca56c2fe12d47" candidate_uuid: "2449fe0928b543e49e272470c621a01a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9080c60e3e440999f4ed4fa4e342d8a" is_pre_election: true
I20260620 23:14:15.226179 25856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1330d571bc65478bb6fca56c2fe12d47" candidate_uuid: "2449fe0928b543e49e272470c621a01a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "444f05061ce9490cb5725fc402922b2d" is_pre_election: true
I20260620 23:14:15.226317 25725 raft_consensus.cc:2454] T 1330d571bc65478bb6fca56c2fe12d47 P f9080c60e3e440999f4ed4fa4e342d8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2449fe0928b543e49e272470c621a01a in term 0.
I20260620 23:14:15.226648 25856 raft_consensus.cc:2454] T 1330d571bc65478bb6fca56c2fe12d47 P 444f05061ce9490cb5725fc402922b2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2449fe0928b543e49e272470c621a01a in term 0.
I20260620 23:14:15.227099 25527 leader_election.cc:302] T 1330d571bc65478bb6fca56c2fe12d47 P 2449fe0928b543e49e272470c621a01a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2449fe0928b543e49e272470c621a01a, f9080c60e3e440999f4ed4fa4e342d8a; no voters:
I20260620 23:14:15.227450 25925 raft_consensus.cc:2781] T 1330d571bc65478bb6fca56c2fe12d47 P 2449fe0928b543e49e272470c621a01a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:14:15.227591 25925 raft_consensus.cc:482] T 1330d571bc65478bb6fca56c2fe12d47 P 2449fe0928b543e49e272470c621a01a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:14:15.227720 25925 raft_consensus.cc:3037] T 1330d571bc65478bb6fca56c2fe12d47 P 2449fe0928b543e49e272470c621a01a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:14:15.229882 25925 raft_consensus.cc:504] T 1330d571bc65478bb6fca56c2fe12d47 P 2449fe0928b543e49e272470c621a01a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9080c60e3e440999f4ed4fa4e342d8a" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 46539 } } peers { permanent_uuid: "444f05061ce9490cb5725fc402922b2d" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46283 } } peers { permanent_uuid: "2449fe0928b543e49e272470c621a01a" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 34215 } }
I20260620 23:14:15.230451 25925 leader_election.cc:288] T 1330d571bc65478bb6fca56c2fe12d47 P 2449fe0928b543e49e272470c621a01a [CANDIDATE]: Term 1 election: Requested vote from peers f9080c60e3e440999f4ed4fa4e342d8a (127.10.191.194:46539), 444f05061ce9490cb5725fc402922b2d (127.10.191.195:46283)
I20260620 23:14:15.230983 25725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1330d571bc65478bb6fca56c2fe12d47" candidate_uuid: "2449fe0928b543e49e272470c621a01a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9080c60e3e440999f4ed4fa4e342d8a"
I20260620 23:14:15.231082 25856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1330d571bc65478bb6fca56c2fe12d47" candidate_uuid: "2449fe0928b543e49e272470c621a01a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "444f05061ce9490cb5725fc402922b2d"
I20260620 23:14:15.231179 25725 raft_consensus.cc:3037] T 1330d571bc65478bb6fca56c2fe12d47 P f9080c60e3e440999f4ed4fa4e342d8a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:14:15.231319 25856 raft_consensus.cc:3037] T 1330d571bc65478bb6fca56c2fe12d47 P 444f05061ce9490cb5725fc402922b2d [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:14:15.233381 25725 raft_consensus.cc:2454] T 1330d571bc65478bb6fca56c2fe12d47 P f9080c60e3e440999f4ed4fa4e342d8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2449fe0928b543e49e272470c621a01a in term 1.
I20260620 23:14:15.233531 25856 raft_consensus.cc:2454] T 1330d571bc65478bb6fca56c2fe12d47 P 444f05061ce9490cb5725fc402922b2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2449fe0928b543e49e272470c621a01a in term 1.
I20260620 23:14:15.233819 25527 leader_election.cc:302] T 1330d571bc65478bb6fca56c2fe12d47 P 2449fe0928b543e49e272470c621a01a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2449fe0928b543e49e272470c621a01a, f9080c60e3e440999f4ed4fa4e342d8a; no voters:
I20260620 23:14:15.234112 25925 raft_consensus.cc:2781] T 1330d571bc65478bb6fca56c2fe12d47 P 2449fe0928b543e49e272470c621a01a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:14:15.234540 25925 raft_consensus.cc:686] T 1330d571bc65478bb6fca56c2fe12d47 P 2449fe0928b543e49e272470c621a01a [term 1 LEADER]: Becoming Leader. State: Replica: 2449fe0928b543e49e272470c621a01a, State: Running, Role: LEADER
I20260620 23:14:15.235052 25925 consensus_queue.cc:237] T 1330d571bc65478bb6fca56c2fe12d47 P 2449fe0928b543e49e272470c621a01a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9080c60e3e440999f4ed4fa4e342d8a" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 46539 } } peers { permanent_uuid: "444f05061ce9490cb5725fc402922b2d" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46283 } } peers { permanent_uuid: "2449fe0928b543e49e272470c621a01a" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 34215 } }
I20260620 23:14:15.239254 25451 catalog_manager.cc:5495] T 1330d571bc65478bb6fca56c2fe12d47 P 2449fe0928b543e49e272470c621a01a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2449fe0928b543e49e272470c621a01a (127.10.191.193). New cstate: current_term: 1 leader_uuid: "2449fe0928b543e49e272470c621a01a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9080c60e3e440999f4ed4fa4e342d8a" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 46539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "444f05061ce9490cb5725fc402922b2d" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2449fe0928b543e49e272470c621a01a" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 34215 } health_report { overall_health: HEALTHY } } }
I20260620 23:14:15.283217 25916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:15.277617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50834 (local address 127.10.191.193:34215)
0620 23:14:15.277847 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:15.277853 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:15.277873 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:14:15.278078 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:15.278082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:15.278160 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:15.278361 (+ 201us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:14:15.278730 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:15.279544 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:15.281095 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:15.281494 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:15.281628 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:15.282106 (+ 478us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:14:15.282605 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:14:15.282763 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:15.282844 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 23:14:15.282948 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 23:14:15.291342 25725 raft_consensus.cc:1261] T 1330d571bc65478bb6fca56c2fe12d47 P f9080c60e3e440999f4ed4fa4e342d8a [term 1 FOLLOWER]: Refusing update from remote peer 2449fe0928b543e49e272470c621a01a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:14:15.291363 25856 raft_consensus.cc:1261] T 1330d571bc65478bb6fca56c2fe12d47 P 444f05061ce9490cb5725fc402922b2d [term 1 FOLLOWER]: Refusing update from remote peer 2449fe0928b543e49e272470c621a01a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:14:15.292253 25928 consensus_queue.cc:1035] T 1330d571bc65478bb6fca56c2fe12d47 P 2449fe0928b543e49e272470c621a01a [LEADER]: Connected to new peer: Peer: permanent_uuid: "444f05061ce9490cb5725fc402922b2d" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:14:15.292582 25925 consensus_queue.cc:1035] T 1330d571bc65478bb6fca56c2fe12d47 P 2449fe0928b543e49e272470c621a01a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9080c60e3e440999f4ed4fa4e342d8a" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 46539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 23:14:15.304441 25902 tablet.cc:2369] T 1330d571bc65478bb6fca56c2fe12d47 P 444f05061ce9490cb5725fc402922b2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:14:15.306455 25933 mvcc.cc:205] Tried to move back new op lower bound from 7299060757663260672 to 7299060757446180864. Current Snapshot: MvccSnapshot[applied={T|T < 7299060757663260672}]
I20260620 23:14:15.317081 25451 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:48400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:14:15.317384 25451 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:48400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:14:15.320720 25451 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f: Sending DeleteTablet for 3 replicas of tablet 1330d571bc65478bb6fca56c2fe12d47
I20260620 23:14:15.321717 25705 tablet_service.cc:1514] Processing DeleteTablet for tablet 1330d571bc65478bb6fca56c2fe12d47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:14:15 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:44824
I20260620 23:14:15.321980 25836 tablet_service.cc:1514] Processing DeleteTablet for tablet 1330d571bc65478bb6fca56c2fe12d47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:14:15 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:58916
I20260620 23:14:15.322144 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 25489
I20260620 23:14:15.322678 25573 tablet_service.cc:1514] Processing DeleteTablet for tablet 1330d571bc65478bb6fca56c2fe12d47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:14:15 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:50820
I20260620 23:14:15.334214 25941 tablet_replica.cc:330] T 1330d571bc65478bb6fca56c2fe12d47 P f9080c60e3e440999f4ed4fa4e342d8a: stopping tablet replica
I20260620 23:14:15.334687 25941 raft_consensus.cc:2229] T 1330d571bc65478bb6fca56c2fe12d47 P f9080c60e3e440999f4ed4fa4e342d8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:14:15.335227 25941 raft_consensus.cc:2258] T 1330d571bc65478bb6fca56c2fe12d47 P f9080c60e3e440999f4ed4fa4e342d8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:14:15.336366 25942 tablet_replica.cc:330] T 1330d571bc65478bb6fca56c2fe12d47 P 444f05061ce9490cb5725fc402922b2d: stopping tablet replica
I20260620 23:14:15.336760 25942 raft_consensus.cc:2229] T 1330d571bc65478bb6fca56c2fe12d47 P 444f05061ce9490cb5725fc402922b2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:14:15.337263 25942 raft_consensus.cc:2258] T 1330d571bc65478bb6fca56c2fe12d47 P 444f05061ce9490cb5725fc402922b2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:14:15.337966 25941 ts_tablet_manager.cc:1905] T 1330d571bc65478bb6fca56c2fe12d47 P f9080c60e3e440999f4ed4fa4e342d8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:14:15.338474 25943 tablet_replica.cc:330] T 1330d571bc65478bb6fca56c2fe12d47 P 2449fe0928b543e49e272470c621a01a: stopping tablet replica
I20260620 23:14:15.338856 25943 raft_consensus.cc:2229] T 1330d571bc65478bb6fca56c2fe12d47 P 2449fe0928b543e49e272470c621a01a [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:14:15.339565 25943 raft_consensus.cc:2258] T 1330d571bc65478bb6fca56c2fe12d47 P 2449fe0928b543e49e272470c621a01a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:14:15.339691 25942 ts_tablet_manager.cc:1905] T 1330d571bc65478bb6fca56c2fe12d47 P 444f05061ce9490cb5725fc402922b2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:14:15.342510 25943 ts_tablet_manager.cc:1905] T 1330d571bc65478bb6fca56c2fe12d47 P 2449fe0928b543e49e272470c621a01a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:14:15.344930 25941 ts_tablet_manager.cc:1918] T 1330d571bc65478bb6fca56c2fe12d47 P f9080c60e3e440999f4ed4fa4e342d8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:14:15.345091 25941 log.cc:1196] T 1330d571bc65478bb6fca56c2fe12d47 P f9080c60e3e440999f4ed4fa4e342d8a: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/wal/wals/1330d571bc65478bb6fca56c2fe12d47
I20260620 23:14:15.345633 25941 ts_tablet_manager.cc:1939] T 1330d571bc65478bb6fca56c2fe12d47 P f9080c60e3e440999f4ed4fa4e342d8a: Deleting consensus metadata
I20260620 23:14:15.347079 25916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:15.323335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50848 (local address 127.10.191.193:34215)
0620 23:14:15.323516 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:15.323522 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:15.323925 (+ 403us) server_negotiation.cc:407] Connection header received
0620 23:14:15.324131 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:15.324137 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:15.324231 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:15.324328 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:15.325688 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:15.327273 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:15.329008 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:15.329880 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:15.330023 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:15.335383 (+ 5360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:15.335402 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:15.335424 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:15.335480 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:15.340742 (+ 5262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:15.342687 (+ 1945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:15.342696 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:15.342700 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:15.342852 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:15.345129 (+ 2277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:15.345210 (+ 81us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:15.345215 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:15.345673 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:15.346081 (+ 408us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:15.346436 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 23:14:15.346684 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 23:14:15.347462 25437 catalog_manager.cc:4841] TS f9080c60e3e440999f4ed4fa4e342d8a (127.10.191.194:46539): tablet 1330d571bc65478bb6fca56c2fe12d47 (table test-table [id=7f3c0e5e57764062903ffa41092a5381]) successfully deleted
I20260620 23:14:15.348548 25633 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:14:15.350180 25942 ts_tablet_manager.cc:1918] T 1330d571bc65478bb6fca56c2fe12d47 P 444f05061ce9490cb5725fc402922b2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:14:15.350358 25942 log.cc:1196] T 1330d571bc65478bb6fca56c2fe12d47 P 444f05061ce9490cb5725fc402922b2d: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/wal/wals/1330d571bc65478bb6fca56c2fe12d47
I20260620 23:14:15.350993 25942 ts_tablet_manager.cc:1939] T 1330d571bc65478bb6fca56c2fe12d47 P 444f05061ce9490cb5725fc402922b2d: Deleting consensus metadata
I20260620 23:14:15.352552 25437 catalog_manager.cc:4841] TS 444f05061ce9490cb5725fc402922b2d (127.10.191.195:46283): tablet 1330d571bc65478bb6fca56c2fe12d47 (table test-table [id=7f3c0e5e57764062903ffa41092a5381]) successfully deleted
I20260620 23:14:15.445941 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 25489
I20260620 23:14:15.463763 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 25642
W20260620 23:14:15.463866 25437 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.191.193:34215 (error 108)
I20260620 23:14:15.465693 25915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:15.465281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:34215 (local address 127.0.0.1:50850)
0620 23:14:15.465452 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:15.465562 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.193:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 23:14:15.465914 25437 catalog_manager.cc:4584] TS 2449fe0928b543e49e272470c621a01a (127.10.191.193:34215): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1330d571bc65478bb6fca56c2fe12d47: Network error: Client connection negotiation failed: client connection to 127.10.191.193:34215: connect: Connection refused (error 111)
I20260620 23:14:15.477517 25912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:15.464562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44826 (local address 127.10.191.194:46539)
0620 23:14:15.464746 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:15.464752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:15.464930 (+ 178us) server_negotiation.cc:407] Connection header received
0620 23:14:15.465142 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:15.465149 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:15.465242 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:15.465353 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:15.466817 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:15.467604 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:15.468898 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:15.469325 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:15.469458 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:15.472195 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:15.472208 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:15.472222 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:15.472253 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:15.474770 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:15.475466 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:15.475475 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:15.475479 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:15.475609 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:15.475984 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:15.475991 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:15.475994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:15.476349 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:15.476640 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:15.476973 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 23:14:15.477185 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 23:14:15.478169 25765 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:14:15.519569 25915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:15.519000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:34215 (local address 127.0.0.1:50864)
0620 23:14:15.519230 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:15.519369 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.193:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 23:14:15.520679 25915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:15.520289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:34215 (local address 127.0.0.1:50878)
0620 23:14:15.520447 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:15.520537 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.193:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 23:14:15.568779 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 25642
I20260620 23:14:15.581480 25915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:15.580823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:34215 (local address 127.0.0.1:50890)
0620 23:14:15.581100 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:15.581225 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.193:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 23:14:15.582705 25915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:15.582236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:34215 (local address 127.0.0.1:50902)
0620 23:14:15.582444 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:15.582545 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.193:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 23:14:15.587015 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 25773
I20260620 23:14:15.600255 25914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:15.587864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58928 (local address 127.10.191.195:46283)
0620 23:14:15.588037 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:15.588042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:15.588060 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:14:15.588198 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:15.588203 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:15.588269 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:15.588355 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:15.589671 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:15.590433 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:15.591807 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:15.592192 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:15.592292 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:15.594751 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:15.594773 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:15.594789 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:15.594825 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:15.597628 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:15.598253 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:15.598260 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:15.598263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:15.598411 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:15.598782 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:15.598786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:15.598788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:15.599266 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:15.599504 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:15.599749 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 23:14:15.599918 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 23:14:15.600888 25896 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:14:15.664131 25915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:15.663585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:34215 (local address 127.0.0.1:50914)
0620 23:14:15.663828 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:15.663939 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.193:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 23:14:15.665185 25915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:15.664774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:34215 (local address 127.0.0.1:50922)
0620 23:14:15.664978 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:15.665061 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.193:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260620 23:14:15.690244 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 25773
I20260620 23:14:15.712977 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 25419
I20260620 23:14:15.714228 25482 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:14:15.829201 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 25419
I20260620 23:14:15.850373 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:43585
--webserver_interface=127.10.191.254
--webserver_port=39291
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:39607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:43585
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:14:16.172035 25949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:14:16.172317 25949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:14:16.172472 25949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:14:16.179690 25949 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:14:16.179790 25949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:14:16.179891 25949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:14:16.179989 25949 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:14:16.180068 25949 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:14:16.189733 25949 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:39607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:43585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:43585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=39291
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:14:16.191082 25949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:14:16.192396 25949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:14:16.204018 25957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:16.204139 25955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:16.204244 25954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:14:16.205097 25949 server_base.cc:1034] running on GCE node
I20260620 23:14:16.205859 25949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:14:16.207285 25949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:14:16.208504 25949 hybrid_clock.cc:648] HybridClock initialized: now 1781997256208467 us; error 32 us; skew 500 ppm
I20260620 23:14:16.208895 25949 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:14:16 dist-test-slave-jkbq krb5kdc[24869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997256, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:14:16.212136 25949 init.cc:377] Logged in from keytab as kudu/127.10.191.254@KRBTEST.COM (short username kudu)
I20260620 23:14:16.213721 25949 webserver.cc:460] Webserver started at http://127.10.191.254:39291/ using document root <none> and password file <none>
I20260620 23:14:16.214417 25949 fs_manager.cc:359] Metadata directory not provided
I20260620 23:14:16.214548 25949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:14:16.220045 25949 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:14:16.222954 25964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:16.224503 25949 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:14:16.224695 25949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "5cc88259af6a46a7be7375becd6aef0f"
format_stamp: "Formatted at 2026-06-20 23:14:10 on dist-test-slave-jkbq"
server_key: "f23fe6202a7f8300f94db4d3a4b13825"
server_key_iv: "61f9b18d91e0633c9112f3abd4516173"
server_key_version: "encryptionkey@0"
I20260620 23:14:16.225709 25949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:14:16.240180 25949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:14:16.244267 25949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:14:16.244552 25949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:14:16.263131 25949 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:43585
I20260620 23:14:16.263156 26015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:43585 every 8 connection(s)
I20260620 23:14:16.264611 25949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:14:16.271023 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 25949
I20260620 23:14:16.271648 26016 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:14:16.272017 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.193:34215
--local_ip_for_outbound_sockets=127.10.191.193
--tserver_master_addrs=127.10.191.254:43585
--webserver_port=37433
--webserver_interface=127.10.191.193
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:39607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:14:16.276858 26016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f: Bootstrap starting.
I20260620 23:14:16.302191 26016 log.cc:824] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f: Log is configured to *not* fsync() on all Append() calls
I20260620 23:14:16.318548 26016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:14:16.319178 26016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f: Bootstrap complete.
I20260620 23:14:16.328801 26016 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc88259af6a46a7be7375becd6aef0f" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43585 } }
I20260620 23:14:16.329754 26016 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cc88259af6a46a7be7375becd6aef0f, State: Initialized, Role: FOLLOWER
I20260620 23:14:16.330384 26016 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc88259af6a46a7be7375becd6aef0f" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43585 } }
I20260620 23:14:16.330660 26016 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:14:16.330783 26016 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:14:16.330988 26016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [term 2 FOLLOWER]: Advancing to term 3
I20260620 23:14:16.334762 26016 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc88259af6a46a7be7375becd6aef0f" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43585 } }
I20260620 23:14:16.335295 26016 leader_election.cc:302] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cc88259af6a46a7be7375becd6aef0f; no voters:
I20260620 23:14:16.335963 26016 leader_election.cc:288] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 23:14:16.336091 26020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [term 3 FOLLOWER]: Leader election won for term 3
I20260620 23:14:16.337212 26020 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [term 3 LEADER]: Becoming Leader. State: Replica: 5cc88259af6a46a7be7375becd6aef0f, State: Running, Role: LEADER
I20260620 23:14:16.337795 26020 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc88259af6a46a7be7375becd6aef0f" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43585 } }
I20260620 23:14:16.338217 26016 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:14:16.340400 26022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cc88259af6a46a7be7375becd6aef0f. Latest consensus state: current_term: 3 leader_uuid: "5cc88259af6a46a7be7375becd6aef0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc88259af6a46a7be7375becd6aef0f" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43585 } } }
I20260620 23:14:16.340523 26021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "5cc88259af6a46a7be7375becd6aef0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cc88259af6a46a7be7375becd6aef0f" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43585 } } }
I20260620 23:14:16.340773 26022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [sys.catalog]: This master's current role is: LEADER
I20260620 23:14:16.340854 26021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cc88259af6a46a7be7375becd6aef0f [sys.catalog]: This master's current role is: LEADER
I20260620 23:14:16.348869 26027 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:14:16.359642 26027 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:14:16.361603 26027 catalog_manager.cc:1259] Loaded cluster ID: b8624621d19c4ecc95b33ee0bd1970fb
I20260620 23:14:16.361729 26027 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 23:14:16.364355 26027 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f03512b59ad google::LogMessage::Fail() at ??:0
@ 0x7f03512b888a google::LogMessage::SendToLog() at ??:0
@ 0x7f03512b54de google::LogMessage::Flush() at ??:0
@ 0x7f03512b6a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0366ce8512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f0366ce5976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f0366d6d610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f0366d6d439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f0353f01345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f0353f0491c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f0353f04799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f0353edfeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f03571b96db start_thread at ??:0
@ 0x7f035014f61f clone at ??:0
W20260620 23:14:16.675079 26018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:14:16.675351 26018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:14:16.675444 26018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:14:16.683051 26018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:14:16.683149 26018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:14:16.683261 26018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.193
I20260620 23:14:16.693415 26018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:39607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.193:34215
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.191.193
--webserver_port=37433
--tserver_master_addrs=127.10.191.254:43585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.193
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:14:16.694594 26018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:14:16.695911 26018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:14:16.708586 26057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:16.708609 26060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:16.708544 26058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:14:16.709698 26018 server_base.cc:1034] running on GCE node
I20260620 23:14:16.710276 26018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:14:16.711328 26018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:14:16.712574 26018 hybrid_clock.cc:648] HybridClock initialized: now 1781997256712513 us; error 52 us; skew 500 ppm
I20260620 23:14:16.713166 26018 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:14:16 dist-test-slave-jkbq krb5kdc[24869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997256, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:14:16.716317 26018 init.cc:377] Logged in from keytab as kudu/127.10.191.193@KRBTEST.COM (short username kudu)
I20260620 23:14:16.717728 26018 webserver.cc:460] Webserver started at http://127.10.191.193:37433/ using document root <none> and password file <none>
I20260620 23:14:16.718355 26018 fs_manager.cc:359] Metadata directory not provided
I20260620 23:14:16.718464 26018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:14:16.723652 26018 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:14:16.726397 26067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:16.727815 26018 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:14:16.727948 26018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/wal
uuid: "2449fe0928b543e49e272470c621a01a"
format_stamp: "Formatted at 2026-06-20 23:14:11 on dist-test-slave-jkbq"
server_key: "0f6302b1228b49492a3c285bcecbdfc4"
server_key_iv: "310538ec03f0d7d0ccfa68a53f874241"
server_key_version: "encryptionkey@0"
I20260620 23:14:16.728826 26018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:14:16.757124 26018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:14:16.761365 26018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:14:16.761593 26018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:14:16.762506 26018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:14:16.764214 26074 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260620 23:14:16.769393 26074 ts_tablet_manager.cc:1852] T 1330d571bc65478bb6fca56c2fe12d47 P 2449fe0928b543e49e272470c621a01a: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260620 23:14:16.770483 26074 ts_tablet_manager.cc:1905] T 1330d571bc65478bb6fca56c2fe12d47 P 2449fe0928b543e49e272470c621a01a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:14:16.777449 26074 ts_tablet_manager.cc:1918] T 1330d571bc65478bb6fca56c2fe12d47 P 2449fe0928b543e49e272470c621a01a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:14:16.777649 26074 log.cc:1196] T 1330d571bc65478bb6fca56c2fe12d47 P 2449fe0928b543e49e272470c621a01a: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/wal/wals/1330d571bc65478bb6fca56c2fe12d47
I20260620 23:14:16.778127 26074 ts_tablet_manager.cc:1939] T 1330d571bc65478bb6fca56c2fe12d47 P 2449fe0928b543e49e272470c621a01a: Deleting consensus metadata
I20260620 23:14:16.778501 26018 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260620 23:14:16.778621 26018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.001s sys 0.000s
I20260620 23:14:16.778769 26018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:14:16.778900 26018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:16.811669 26018 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.193:34215
I20260620 23:14:16.811753 26180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.193:34215 every 8 connection(s)
I20260620 23:14:16.813086 26018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
I20260620 23:14:16.818084 26183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:16.816229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.193:45597)
0620 23:14:16.816984 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:16.817219 (+ 235us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":297,"threads_started":1}
W20260620 23:14:16.818708 26181 heartbeater.cc:644] Failed to heartbeat to 127.10.191.254:43585 (0 consecutive failures): Network error: Failed to ping master at 127.10.191.254:43585: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
I20260620 23:14:16.819545 26183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:16.819229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.193:55021)
0620 23:14:16.819374 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:16.819449 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 23:14:16.820367 26183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:16.820035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.193:43085)
0620 23:14:16.820172 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:16.820252 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 23:14:16.820555 26181 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 23:14:16.821182 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 26018
I20260620 23:14:16.821789 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.194:46539
--local_ip_for_outbound_sockets=127.10.191.194
--tserver_master_addrs=127.10.191.254:43585
--webserver_port=35681
--webserver_interface=127.10.191.194
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:39607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:14:17.148126 26184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:14:17.148399 26184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:14:17.148494 26184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:14:17.155476 26184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:14:17.155573 26184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:14:17.155682 26184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.194
I20260620 23:14:17.165066 26184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:39607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.194:46539
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.191.194
--webserver_port=35681
--tserver_master_addrs=127.10.191.254:43585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.194
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:14:17.166225 26184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:14:17.167487 26184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:14:17.179940 26192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:17.180022 26189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:17.179996 26190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:14:17.181533 26184 server_base.cc:1034] running on GCE node
I20260620 23:14:17.182116 26184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:14:17.183080 26184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:14:17.184299 26184 hybrid_clock.cc:648] HybridClock initialized: now 1781997257184269 us; error 42 us; skew 500 ppm
I20260620 23:14:17.184607 26184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:14:17 dist-test-slave-jkbq krb5kdc[24869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997257, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:14:17.187587 26184 init.cc:377] Logged in from keytab as kudu/127.10.191.194@KRBTEST.COM (short username kudu)
I20260620 23:14:17.188926 26184 webserver.cc:460] Webserver started at http://127.10.191.194:35681/ using document root <none> and password file <none>
I20260620 23:14:17.189595 26184 fs_manager.cc:359] Metadata directory not provided
I20260620 23:14:17.189738 26184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:14:17.194986 26184 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:14:17.197707 26199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:17.199107 26184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:14:17.199281 26184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/wal
uuid: "f9080c60e3e440999f4ed4fa4e342d8a"
format_stamp: "Formatted at 2026-06-20 23:14:11 on dist-test-slave-jkbq"
server_key: "f18e0ba4d25707f3a5b5713a61a69a62"
server_key_iv: "36314dcd557976ff2644feb0dc00c42d"
server_key_version: "encryptionkey@0"
I20260620 23:14:17.200244 26184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:14:17.217084 26184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:14:17.220893 26184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:14:17.221158 26184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:14:17.222095 26184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:14:17.223652 26184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:14:17.223768 26184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:17.223899 26184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:14:17.223994 26184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:17.256484 26184 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.194:46539
I20260620 23:14:17.256575 26311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.194:46539 every 8 connection(s)
I20260620 23:14:17.257946 26184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
I20260620 23:14:17.262467 26314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:17.260711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.194:36371)
0620 23:14:17.261416 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:17.261609 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":273,"threads_started":1}
W20260620 23:14:17.263075 26312 heartbeater.cc:644] Failed to heartbeat to 127.10.191.254:43585 (0 consecutive failures): Network error: Failed to ping master at 127.10.191.254:43585: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
I20260620 23:14:17.263815 26314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:17.263531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.194:49419)
0620 23:14:17.263655 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:17.263724 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 23:14:17.264582 26314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:17.264306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.194:32925)
0620 23:14:17.264422 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:17.264488 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 23:14:17.264916 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 26184
W20260620 23:14:17.265183 26312 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 23:14:17.265583 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.195:46283
--local_ip_for_outbound_sockets=127.10.191.195
--tserver_master_addrs=127.10.191.254:43585
--webserver_port=41539
--webserver_interface=127.10.191.195
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:39607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:14:17.578158 26315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:14:17.578445 26315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:14:17.578617 26315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:14:17.585677 26315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:14:17.585812 26315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:14:17.585995 26315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.195
I20260620 23:14:17.595332 26315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:39607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.195:46283
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.191.195
--webserver_port=41539
--tserver_master_addrs=127.10.191.254:43585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.195
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:14:17.596558 26315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:14:17.597841 26315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:14:17.610496 26321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:17.610496 26320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:17.610504 26323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:14:17.611514 26315 server_base.cc:1034] running on GCE node
I20260620 23:14:17.612077 26315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:14:17.612942 26315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:14:17.614149 26315 hybrid_clock.cc:648] HybridClock initialized: now 1781997257614104 us; error 58 us; skew 500 ppm
I20260620 23:14:17.614475 26315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:14:17 dist-test-slave-jkbq krb5kdc[24869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997257, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:14:17.622558 26315 init.cc:377] Logged in from keytab as kudu/127.10.191.195@KRBTEST.COM (short username kudu)
I20260620 23:14:17.624135 26315 webserver.cc:460] Webserver started at http://127.10.191.195:41539/ using document root <none> and password file <none>
I20260620 23:14:17.624859 26315 fs_manager.cc:359] Metadata directory not provided
I20260620 23:14:17.625007 26315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:14:17.630167 26315 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 23:14:17.632895 26330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:17.634243 26315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:14:17.634413 26315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/wal
uuid: "444f05061ce9490cb5725fc402922b2d"
format_stamp: "Formatted at 2026-06-20 23:14:12 on dist-test-slave-jkbq"
server_key: "08ea9c8bc3c44b03c0bc58960f40e04e"
server_key_iv: "3efdd9e69e74f58bfdffe17f4820d804"
server_key_version: "encryptionkey@0"
I20260620 23:14:17.635471 26315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:14:17.657393 26315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:14:17.661664 26315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:14:17.661947 26315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:14:17.662998 26315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:14:17.664311 26315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:14:17.664438 26315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:17.664575 26315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:14:17.664664 26315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:17.697544 26315 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.195:46283
I20260620 23:14:17.697629 26442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.195:46283 every 8 connection(s)
I20260620 23:14:17.699035 26315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
I20260620 23:14:17.703758 26445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:17.701836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.195:51677)
0620 23:14:17.702549 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:17.702757 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":261,"threads_started":1}
W20260620 23:14:17.704373 26443 heartbeater.cc:644] Failed to heartbeat to 127.10.191.254:43585 (0 consecutive failures): Network error: Failed to ping master at 127.10.191.254:43585: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
I20260620 23:14:17.705135 26445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:17.704825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.195:60301)
0620 23:14:17.704979 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:17.705041 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 23:14:17.705880 26445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:17.705586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.195:41311)
0620 23:14:17.705704 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:17.705780 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 23:14:17.706333 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 26315
W20260620 23:14:17.706634 26443 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 23:14:17.821821 26447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:17.821023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.193:32791)
0620 23:14:17.821482 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:17.821593 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":223,"threads_started":1}
I20260620 23:14:18.266933 26448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:18.265967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.194:58677)
0620 23:14:18.266522 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:18.266654 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":267,"threads_started":1}
I20260620 23:14:18.707926 26449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:18.707160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.195:55953)
0620 23:14:18.707592 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:18.707713 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":192,"threads_started":1}
I20260620 23:14:18.823421 26450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:18.822655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.193:41393)
0620 23:14:18.823141 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:18.823229 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":287,"threads_started":1}
I20260620 23:14:19.268522 26451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:19.267776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.194:42401)
0620 23:14:19.268264 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:19.268344 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":247,"threads_started":1}
I20260620 23:14:19.709575 26452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:19.708820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.195:53793)
0620 23:14:19.709330 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:19.709424 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":279,"threads_started":1}
I20260620 23:14:19.824962 26453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:19.824231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.193:42183)
0620 23:14:19.824686 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:19.824792 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":235,"threads_started":1}
I20260620 23:14:20.270149 26454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:20.269444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.194:41783)
0620 23:14:20.269901 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:20.269983 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":240,"threads_started":1}
I20260620 23:14:20.711226 26455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:20.710475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.195:42847)
0620 23:14:20.710974 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:20.711054 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":217,"threads_started":1}
I20260620 23:14:20.826424 26456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:20.825766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.193:52709)
0620 23:14:20.826193 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:20.826269 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":221,"threads_started":1}
I20260620 23:14:21.271880 26457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:21.271122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.194:36549)
0620 23:14:21.271601 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:21.271683 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":237,"threads_started":1}
I20260620 23:14:21.712749 26458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:21.712088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.195:44737)
0620 23:14:21.712513 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:21.712592 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":227,"threads_started":1}
I20260620 23:14:21.828071 26459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:21.827321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.193:37919)
0620 23:14:21.827811 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:21.827895 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":249,"threads_started":1}
I20260620 23:14:22.273466 26460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:22.272753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.194:41263)
0620 23:14:22.273189 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:22.273266 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":217,"threads_started":1}
I20260620 23:14:22.714546 26461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:22.713770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.195:34685)
0620 23:14:22.714299 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:22.714385 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":307,"threads_started":1}
I20260620 23:14:22.829627 26462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:22.828948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.193:59115)
0620 23:14:22.829376 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:22.829473 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":221,"threads_started":1}
I20260620 23:14:23.275182 26463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:23.274404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.194:46117)
0620 23:14:23.274843 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:23.275010 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":223,"threads_started":1}
I20260620 23:14:23.716109 26464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:23.715406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.195:56321)
0620 23:14:23.715857 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:23.715949 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":232,"threads_started":1}
I20260620 23:14:23.831346 26465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:23.830515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.193:53635)
0620 23:14:23.831057 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:23.831139 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":228,"threads_started":1}
I20260620 23:14:24.277120 26466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:24.276194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.194:55275)
0620 23:14:24.276750 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:24.276836 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":272,"threads_started":1}
I20260620 23:14:24.717866 26467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:24.717102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.195:51593)
0620 23:14:24.717579 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:24.717667 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":238,"threads_started":1}
I20260620 23:14:24.832964 26468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:24.832190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.193:51645)
0620 23:14:24.832675 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:24.832781 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":267,"threads_started":1}
I20260620 23:14:25.278756 26469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:25.277977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.194:47613)
0620 23:14:25.278437 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:25.278559 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":236,"threads_started":1}
I20260620 23:14:25.719681 26470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:25.718805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.195:55381)
0620 23:14:25.719370 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:25.719490 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":319,"threads_started":1}
I20260620 23:14:25.834524 26471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:25.833796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.193:60879)
0620 23:14:25.834250 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:25.834351 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":227,"threads_started":1}
I20260620 23:14:26.280556 26472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:26.279803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.194:40497)
0620 23:14:26.280282 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:26.280377 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":235,"threads_started":1}
I20260620 23:14:26.721421 26473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:26.720643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.195:57105)
0620 23:14:26.721137 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:26.721218 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":259,"threads_started":1}
I20260620 23:14:26.836300 26474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:26.835510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.193:60139)
0620 23:14:26.836050 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:26.836135 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":311,"threads_started":1}
I20260620 23:14:27.282176 26475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:27.281501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.194:49171)
0620 23:14:27.281921 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:27.282013 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":231,"threads_started":1}
I20260620 23:14:27.723122 26476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:27.722359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.195:48267)
0620 23:14:27.722810 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:27.722959 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":256,"threads_started":1}
I20260620 23:14:27.837992 26477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:27.837225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.193:44639)
0620 23:14:27.837704 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:27.837797 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":263,"threads_started":1}
I20260620 23:14:28.284467 26478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:28.283320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.194:57449)
0620 23:14:28.284027 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:28.284179 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":356,"threads_started":1}
I20260620 23:14:28.724814 26479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:28.724028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.195:60123)
0620 23:14:28.724537 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:28.724632 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":227,"threads_started":1}
I20260620 23:14:28.839815 26480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:28.838987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.193:52681)
0620 23:14:28.839467 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:28.839607 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":241,"threads_started":1}
I20260620 23:14:29.286089 26481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:29.285411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.194:54029)
0620 23:14:29.285849 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:29.285931 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":219,"threads_started":1}
I20260620 23:14:29.726410 26482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:29.725715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.195:32973)
0620 23:14:29.726176 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:29.726255 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":257,"threads_started":1}
I20260620 23:14:29.841634 26483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:29.840792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.193:56873)
0620 23:14:29.841343 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:29.841444 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":294,"threads_started":1}
I20260620 23:14:30.289124 26484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:30.287948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.194:49807)
0620 23:14:30.288716 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:30.288850 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":473,"threads_started":1}
I20260620 23:14:30.728039 26485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:30.727312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.195:45853)
0620 23:14:30.727756 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:30.727866 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":228,"threads_started":1}
I20260620 23:14:30.843286 26486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:30.842540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.193:39503)
0620 23:14:30.843021 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:30.843105 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":203,"threads_started":1}
I20260620 23:14:31.290841 26487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:31.290122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.194:51881)
0620 23:14:31.290583 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:31.290663 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":225,"threads_started":1}
I20260620 23:14:31.729769 26488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:31.729019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.195:49421)
0620 23:14:31.729518 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:31.729595 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":238,"threads_started":1}
I20260620 23:14:31.845149 26489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:31.844320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.193:40523)
0620 23:14:31.844861 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:31.844970 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":302,"threads_started":1}
I20260620 23:14:32.292788 26490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:32.292058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.194:35359)
0620 23:14:32.292507 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:32.292602 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":223,"threads_started":1}
I20260620 23:14:32.708925 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 26018
I20260620 23:14:32.726691 26491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:32.709939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39254 (local address 127.10.191.193:34215)
0620 23:14:32.710577 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:32.710590 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:32.710628 (+ 38us) server_negotiation.cc:407] Connection header received
0620 23:14:32.712503 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:32.712514 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:32.712960 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:32.713260 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:32.714574 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:32.715897 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:32.717144 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:32.717845 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:32.717984 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:32.720686 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:32.720694 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:32.720709 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:32.720747 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:32.723790 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:32.724412 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:32.724419 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:32.724425 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:32.724587 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:32.724924 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:32.724927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:32.724929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:32.725479 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:32.725717 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:32.726039 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 23:14:32.726390 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":324,"threads_started":1}
I20260620 23:14:32.728152 26176 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:14:32.731700 26493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:32.730855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:43585 (local address 127.10.191.195:53881)
0620 23:14:32.731337 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:32.731484 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":238,"threads_started":1}
I20260620 23:14:32.812851 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 26018
I20260620 23:14:32.829963 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 26184
I20260620 23:14:32.847740 26494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:32.830983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49808 (local address 127.10.191.194:46539)
0620 23:14:32.831459 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:32.831473 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:32.831510 (+ 37us) server_negotiation.cc:407] Connection header received
0620 23:14:32.832962 (+ 1452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:32.832971 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:32.833393 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:32.833787 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:32.835329 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:32.836564 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:32.837914 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:32.838703 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:32.838940 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:32.841704 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:32.841712 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:32.841727 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:32.841763 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:32.844684 (+ 2921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:32.845339 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:32.845346 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:32.845353 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:32.845546 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:32.845859 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:32.845862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:32.845864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:32.846356 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:32.846607 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:32.847054 (+ 447us) server_negotiation.cc:299] Negotiation successful
0620 23:14:32.847451 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":200,"threads_started":1}
I20260620 23:14:32.849195 26307 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:14:32.932439 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 26184
I20260620 23:14:32.950367 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 26315
I20260620 23:14:32.968412 26496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:32.951608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55442 (local address 127.10.191.195:46283)
0620 23:14:32.952318 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:32.952331 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:32.952373 (+ 42us) server_negotiation.cc:407] Connection header received
0620 23:14:32.953964 (+ 1591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:32.953973 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:32.954413 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:32.954705 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:32.956093 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:32.957354 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:32.958577 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:32.959366 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:32.959507 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:32.962128 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:32.962136 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:32.962151 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:32.962188 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:32.965165 (+ 2977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:32.965955 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:32.965962 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:32.965969 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:32.966130 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:32.966488 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:32.966491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:32.966493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:32.967090 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:32.967323 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:32.967663 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 23:14:32.968109 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":368,"threads_started":1}
I20260620 23:14:32.969761 26438 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:14:33.052078 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 26315
2026-06-20T23:14:33Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (23912 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:14:33 dist-test-slave-jkbq krb5kdc[26500](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:14:33 dist-test-slave-jkbq krb5kdc[26500](info): set up 2 sockets
Jun 20 23:14:33 dist-test-slave-jkbq krb5kdc[26500](info): commencing operation
krb5kdc: starting...
W20260620 23:14:34.155447 11007 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Jun 20 23:14:34 dist-test-slave-jkbq krb5kdc[26500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:14:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:14:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.191.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
I20260620 23:14:34.390658 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:32873
--webserver_interface=127.10.191.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:33627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:32873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:14:34.718007 26513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:14:34.718302 26513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:14:34.718392 26513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:14:34.725639 26513 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 23:14:34.725741 26513 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 23:14:34.725780 26513 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 23:14:34.725823 26513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:14:34.725878 26513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:14:34.725947 26513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:14:34.725993 26513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:14:34.726040 26513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:14:34.735824 26513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:33627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:32873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:32873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:14:34.737056 26513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:14:34.738454 26513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:14:34.750648 26518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:34.750591 26519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:34.750671 26521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:14:34.751880 26513 server_base.cc:1034] running on GCE node
I20260620 23:14:34.752658 26513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:14:34.753975 26513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:14:34.755191 26513 hybrid_clock.cc:648] HybridClock initialized: now 1781997274755131 us; error 39 us; skew 500 ppm
I20260620 23:14:34.755488 26513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:14:34 dist-test-slave-jkbq krb5kdc[26500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997274, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:14:34.758591 26513 init.cc:377] Logged in from keytab as kudu/127.10.191.254@KRBTEST.COM (short username kudu)
I20260620 23:14:34.760243 26513 webserver.cc:460] Webserver started at http://127.10.191.254:34225/ using document root <none> and password file <none>
I20260620 23:14:34.760978 26513 fs_manager.cc:359] Metadata directory not provided
I20260620 23:14:34.761106 26513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:14:34.761448 26513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:14:34.764487 26513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "d6d9ed6d31094bb29fa59de15a5924a3"
format_stamp: "Formatted at 2026-06-20 23:14:34 on dist-test-slave-jkbq"
server_key: "2cf0710873a8459633bae0f7d6613168"
server_key_iv: "655b8f7dd28bd835c94f275f04379e4f"
server_key_version: "encryptionkey@0"
I20260620 23:14:34.765251 26513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "d6d9ed6d31094bb29fa59de15a5924a3"
format_stamp: "Formatted at 2026-06-20 23:14:34 on dist-test-slave-jkbq"
server_key: "2cf0710873a8459633bae0f7d6613168"
server_key_iv: "655b8f7dd28bd835c94f275f04379e4f"
server_key_version: "encryptionkey@0"
I20260620 23:14:34.770366 26513 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 23:14:34.773819 26528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:34.775316 26513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:14:34.775502 26513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "d6d9ed6d31094bb29fa59de15a5924a3"
format_stamp: "Formatted at 2026-06-20 23:14:34 on dist-test-slave-jkbq"
server_key: "2cf0710873a8459633bae0f7d6613168"
server_key_iv: "655b8f7dd28bd835c94f275f04379e4f"
server_key_version: "encryptionkey@0"
I20260620 23:14:34.775718 26513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:14:34.813055 26513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:14:34.817299 26513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:14:34.817579 26513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:14:34.836781 26513 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:32873
I20260620 23:14:34.836792 26579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:32873 every 8 connection(s)
I20260620 23:14:34.838199 26513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:14:34.842105 26580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:14:34.844014 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 26513
I20260620 23:14:34.844244 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:14:34.844923 11007 external_mini_cluster.cc:1428] Setting key 06da5b2259826fbc1990caddfc4b1b42
I20260620 23:14:34.855464 26580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3: Bootstrap starting.
I20260620 23:14:34.859015 26580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3: Neither blocks nor log segments found. Creating new log.
I20260620 23:14:34.860034 26580 log.cc:824] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3: Log is configured to *not* fsync() on all Append() calls
Jun 20 23:14:34 dist-test-slave-jkbq krb5kdc[26500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:14:34.863198 26580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3: No bootstrap required, opened a new log
I20260620 23:14:34.869853 26583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:34.847094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34280 (local address 127.10.191.254:32873)
0620 23:14:34.847914 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:34.847936 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:34.847993 (+ 57us) server_negotiation.cc:407] Connection header received
0620 23:14:34.850401 (+ 2408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:34.850431 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:34.850974 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:34.851428 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:34.852924 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:34.854906 (+ 1982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:34.856338 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:34.857427 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:34.857624 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:34.862390 (+ 4766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:34.862398 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:34.862412 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:34.862469 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:34.865649 (+ 3180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:34.866517 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:34.866523 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:34.866529 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:34.866703 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:34.867136 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:34.867139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:34.867141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:34.867687 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:34.867904 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:34.868391 (+ 487us) server_negotiation.cc:299] Negotiation successful
0620 23:14:34.868768 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":333,"threads_started":1}
I20260620 23:14:34.872448 26580 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d9ed6d31094bb29fa59de15a5924a3" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 32873 } }
I20260620 23:14:34.872900 26580 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:14:34.873039 26580 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6d9ed6d31094bb29fa59de15a5924a3, State: Initialized, Role: FOLLOWER
I20260620 23:14:34.873595 26580 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d9ed6d31094bb29fa59de15a5924a3" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 32873 } }
I20260620 23:14:34.873842 26580 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:14:34.873972 26580 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:14:34.874145 26580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:14:34.876533 26580 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d9ed6d31094bb29fa59de15a5924a3" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 32873 } }
I20260620 23:14:34.877023 26580 leader_election.cc:302] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6d9ed6d31094bb29fa59de15a5924a3; no voters:
I20260620 23:14:34.877583 26580 leader_election.cc:288] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:14:34.877728 26585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:14:34.878399 26585 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [term 1 LEADER]: Becoming Leader. State: Replica: d6d9ed6d31094bb29fa59de15a5924a3, State: Running, Role: LEADER
I20260620 23:14:34.878997 26585 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d9ed6d31094bb29fa59de15a5924a3" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 32873 } }
I20260620 23:14:34.879815 26580 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:14:34.882416 26587 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6d9ed6d31094bb29fa59de15a5924a3. Latest consensus state: current_term: 1 leader_uuid: "d6d9ed6d31094bb29fa59de15a5924a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d9ed6d31094bb29fa59de15a5924a3" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 32873 } } }
I20260620 23:14:34.883002 26587 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [sys.catalog]: This master's current role is: LEADER
I20260620 23:14:34.882587 26586 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6d9ed6d31094bb29fa59de15a5924a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d9ed6d31094bb29fa59de15a5924a3" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 32873 } } }
I20260620 23:14:34.883608 26586 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [sys.catalog]: This master's current role is: LEADER
I20260620 23:14:34.884881 26593 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:14:34.890410 26593 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:14:34.899709 26593 catalog_manager.cc:1347] Generated new cluster ID: a365fdc97ce44b7aaa187d49cea26635
I20260620 23:14:34.899842 26593 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:14:34.925465 26593 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:14:34.926856 26593 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:14:34.937901 26593 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3: Generated new TSK 0
I20260620 23:14:34.938741 26593 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.191.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab.
Entry for principal kudu/127.10.191.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab.
I20260620 23:14:35.010597 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.193:0
--local_ip_for_outbound_sockets=127.10.191.193
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:32873
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:33627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:14:35.340209 26606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:14:35.340482 26606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:14:35.340639 26606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:14:35.348052 26606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:14:35.348157 26606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:14:35.348366 26606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.193
I20260620 23:14:35.358158 26606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:33627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:32873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.193
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:14:35.359552 26606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:14:35.360821 26606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:14:35.373989 26611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:35.373963 26612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:35.374056 26614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:14:35.374980 26606 server_base.cc:1034] running on GCE node
I20260620 23:14:35.375622 26606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:14:35.376541 26606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:14:35.377775 26606 hybrid_clock.cc:648] HybridClock initialized: now 1781997275377728 us; error 33 us; skew 500 ppm
I20260620 23:14:35.378082 26606 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:14:35 dist-test-slave-jkbq krb5kdc[26500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997275, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:14:35.384788 26606 init.cc:377] Logged in from keytab as kudu/127.10.191.193@KRBTEST.COM (short username kudu)
I20260620 23:14:35.386380 26606 webserver.cc:460] Webserver started at http://127.10.191.193:45075/ using document root <none> and password file <none>
I20260620 23:14:35.387171 26606 fs_manager.cc:359] Metadata directory not provided
I20260620 23:14:35.387315 26606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:14:35.387622 26606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:14:35.390550 26606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/data/instance:
uuid: "c5ff4a994ed24d10ae1b3b1f8fef3789"
format_stamp: "Formatted at 2026-06-20 23:14:35 on dist-test-slave-jkbq"
server_key: "eb9a2a15ac4996c235cb9c64bf327582"
server_key_iv: "85a36910358a037b75d27b2804e55630"
server_key_version: "encryptionkey@0"
I20260620 23:14:35.391343 26606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance:
uuid: "c5ff4a994ed24d10ae1b3b1f8fef3789"
format_stamp: "Formatted at 2026-06-20 23:14:35 on dist-test-slave-jkbq"
server_key: "eb9a2a15ac4996c235cb9c64bf327582"
server_key_iv: "85a36910358a037b75d27b2804e55630"
server_key_version: "encryptionkey@0"
I20260620 23:14:35.396482 26606 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 23:14:35.399940 26621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:35.401262 26606 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:14:35.401453 26606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/wal
uuid: "c5ff4a994ed24d10ae1b3b1f8fef3789"
format_stamp: "Formatted at 2026-06-20 23:14:35 on dist-test-slave-jkbq"
server_key: "eb9a2a15ac4996c235cb9c64bf327582"
server_key_iv: "85a36910358a037b75d27b2804e55630"
server_key_version: "encryptionkey@0"
I20260620 23:14:35.401656 26606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:14:35.421203 26606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:14:35.425153 26606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:14:35.425447 26606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:14:35.426385 26606 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:14:35.427767 26606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:14:35.427878 26606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:35.428032 26606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:14:35.428105 26606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:35.460254 26606 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.193:33111
I20260620 23:14:35.460340 26733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.193:33111 every 8 connection(s)
I20260620 23:14:35.461532 26606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
I20260620 23:14:35.466696 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 26606
I20260620 23:14:35.466956 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance
I20260620 23:14:35.467746 11007 external_mini_cluster.cc:1428] Setting key c1b0003f8663bce81fe1b64e95185fa8
Jun 20 23:14:35 dist-test-slave-jkbq krb5kdc[26500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997275, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:14:35.483764 26736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:35.464121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:40491 (local address 127.10.191.254:32873)
0620 23:14:35.464566 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:35.464571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:35.465551 (+ 980us) server_negotiation.cc:407] Connection header received
0620 23:14:35.468079 (+ 2528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:35.468084 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:35.468152 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:35.468240 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:35.470035 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:35.470758 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:35.473288 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:35.473674 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:35.473777 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:35.477854 (+ 4077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:35.477859 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:35.477862 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:35.477895 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:35.480740 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:35.481490 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:35.481495 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:35.481500 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:35.481619 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:35.482033 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:35.482038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:35.482041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:35.482307 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:35.482475 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:35.483179 (+ 704us) server_negotiation.cc:299] Negotiation successful
0620 23:14:35.483389 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":229,"threads_started":1}
I20260620 23:14:35.484853 26737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:35.464715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.193:40491)
0620 23:14:35.465388 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:35.465429 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:35.467741 (+ 2312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:35.468502 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:35.468510 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:35.469138 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:14:35.469861 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:35.469887 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:35.470956 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:35.470964 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:35.473123 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:35.473141 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:35.473914 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:35.473922 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:35.474236 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:35.474995 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:14:35.475035 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:14:35.477628 (+ 2593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:14:35.480897 (+ 3269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:35.480906 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:35.480922 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:35.481346 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:35.481725 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:35.481731 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:35.481734 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:35.481918 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:35.482436 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:14:35.482443 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:14:35.482765 (+ 322us) client_negotiation.cc:769] Sending connection context
0620 23:14:35.483079 (+ 314us) client_negotiation.cc:240] Negotiation successful
0620 23:14:35.483592 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":263,"threads_started":1}
I20260620 23:14:35.486363 26734 heartbeater.cc:347] Connected to a master server at 127.10.191.254:32873
I20260620 23:14:35.486680 26734 heartbeater.cc:464] Registering TS with master...
I20260620 23:14:35.487578 26734 heartbeater.cc:511] Master 127.10.191.254:32873 requested a full tablet report, sending...
I20260620 23:14:35.490155 26545 ts_manager.cc:194] Registered new tserver with Master: c5ff4a994ed24d10ae1b3b1f8fef3789 (127.10.191.193:33111)
WARNING: no policy specified for kudu/127.10.191.194@KRBTEST.COM; defaulting to no policy
I20260620 23:14:35.492377 26545 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.193@KRBTEST.COM'} at 127.10.191.193:40491
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab.
Entry for principal kudu/127.10.191.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab.
I20260620 23:14:35.522001 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.194:0
--local_ip_for_outbound_sockets=127.10.191.194
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:32873
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:33627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:14:35.846135 26740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:14:35.846412 26740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:14:35.846568 26740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:14:35.854019 26740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:14:35.854123 26740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:14:35.854285 26740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.194
I20260620 23:14:35.864115 26740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:33627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:32873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.194
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:14:35.865537 26740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:14:35.866945 26740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:14:35.881112 26745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:35.881163 26746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:35.881078 26748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:14:35.881971 26740 server_base.cc:1034] running on GCE node
I20260620 23:14:35.882632 26740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:14:35.883667 26740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:14:35.884872 26740 hybrid_clock.cc:648] HybridClock initialized: now 1781997275884796 us; error 61 us; skew 500 ppm
I20260620 23:14:35.885211 26740 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:14:35 dist-test-slave-jkbq krb5kdc[26500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997275, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:14:35.888538 26740 init.cc:377] Logged in from keytab as kudu/127.10.191.194@KRBTEST.COM (short username kudu)
I20260620 23:14:35.889966 26740 webserver.cc:460] Webserver started at http://127.10.191.194:40087/ using document root <none> and password file <none>
I20260620 23:14:35.890643 26740 fs_manager.cc:359] Metadata directory not provided
I20260620 23:14:35.890787 26740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:14:35.891124 26740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:14:35.893944 26740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/data/instance:
uuid: "30bbb7d7a0764e89aaf87e5609420583"
format_stamp: "Formatted at 2026-06-20 23:14:35 on dist-test-slave-jkbq"
server_key: "283a04a8e4f55dee6a1fe0c77108acf8"
server_key_iv: "931086b80c320db55475310070aa1715"
server_key_version: "encryptionkey@0"
I20260620 23:14:35.894683 26740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance:
uuid: "30bbb7d7a0764e89aaf87e5609420583"
format_stamp: "Formatted at 2026-06-20 23:14:35 on dist-test-slave-jkbq"
server_key: "283a04a8e4f55dee6a1fe0c77108acf8"
server_key_iv: "931086b80c320db55475310070aa1715"
server_key_version: "encryptionkey@0"
I20260620 23:14:35.899721 26740 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260620 23:14:35.903169 26755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:35.904462 26740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:14:35.904680 26740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/wal
uuid: "30bbb7d7a0764e89aaf87e5609420583"
format_stamp: "Formatted at 2026-06-20 23:14:35 on dist-test-slave-jkbq"
server_key: "283a04a8e4f55dee6a1fe0c77108acf8"
server_key_iv: "931086b80c320db55475310070aa1715"
server_key_version: "encryptionkey@0"
I20260620 23:14:35.904907 26740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:14:35.923394 26740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:14:35.927735 26740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:14:35.928033 26740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:14:35.929023 26740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:14:35.930356 26740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:14:35.930467 26740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:35.930624 26740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:14:35.930696 26740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:35.963645 26740 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.194:37623
I20260620 23:14:35.963690 26867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.194:37623 every 8 connection(s)
I20260620 23:14:35.964923 26740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
I20260620 23:14:35.973671 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 26740
I20260620 23:14:35.973887 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance
I20260620 23:14:35.974488 11007 external_mini_cluster.cc:1428] Setting key 02102e82cedf77c44035caed5b2286d2
Jun 20 23:14:35 dist-test-slave-jkbq krb5kdc[26500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997275, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:14:35.986605 26736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:35.967457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:56985 (local address 127.10.191.254:32873)
0620 23:14:35.967635 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:35.967639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:35.968850 (+ 1211us) server_negotiation.cc:407] Connection header received
0620 23:14:35.970599 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:35.970603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:35.970670 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:35.970758 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:35.972195 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:35.972876 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:35.975123 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:35.975667 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:35.975810 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:35.979404 (+ 3594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:35.979413 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:35.979420 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:35.979466 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:35.983174 (+ 3708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:35.983942 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:35.983956 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:35.983963 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:35.984126 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:35.984681 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:35.984690 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:35.984694 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:35.985105 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:35.985317 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:35.986013 (+ 696us) server_negotiation.cc:299] Negotiation successful
0620 23:14:35.986250 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:14:35.987658 26870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:35.967971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.194:56985)
0620 23:14:35.968637 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:35.968709 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:35.970359 (+ 1650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:35.971014 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:35.971023 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:35.971510 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:14:35.972048 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:35.972064 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:35.973026 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:35.973030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:35.974962 (+ 1932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:35.974977 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:35.975894 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:35.975901 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:35.976136 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:35.976891 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:14:35.976920 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:14:35.979179 (+ 2259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:14:35.983346 (+ 4167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:35.983353 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:35.983366 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:35.983801 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:35.984272 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:35.984281 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:35.984287 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:35.984533 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:35.985236 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:14:35.985242 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:14:35.985615 (+ 373us) client_negotiation.cc:769] Sending connection context
0620 23:14:35.985916 (+ 301us) client_negotiation.cc:240] Negotiation successful
0620 23:14:35.986453 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":257,"threads_started":1}
I20260620 23:14:35.989166 26868 heartbeater.cc:347] Connected to a master server at 127.10.191.254:32873
I20260620 23:14:35.989465 26868 heartbeater.cc:464] Registering TS with master...
I20260620 23:14:35.990199 26868 heartbeater.cc:511] Master 127.10.191.254:32873 requested a full tablet report, sending...
I20260620 23:14:35.991616 26545 ts_manager.cc:194] Registered new tserver with Master: 30bbb7d7a0764e89aaf87e5609420583 (127.10.191.194:37623)
I20260620 23:14:35.992424 26545 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.194@KRBTEST.COM'} at 127.10.191.194:56985
WARNING: no policy specified for kudu/127.10.191.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab.
Entry for principal kudu/127.10.191.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab.
I20260620 23:14:36.028393 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.195:0
--local_ip_for_outbound_sockets=127.10.191.195
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:32873
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:33627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:14:36.350121 26873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:14:36.350451 26873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:14:36.350616 26873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:14:36.357942 26873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:14:36.358114 26873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:14:36.358304 26873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.195
I20260620 23:14:36.368208 26873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:33627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:32873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.195
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:14:36.369583 26873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:14:36.370915 26873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:14:36.384647 26878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:36.384756 26879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:36.384727 26881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:14:36.385721 26873 server_base.cc:1034] running on GCE node
I20260620 23:14:36.386309 26873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:14:36.387290 26873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:14:36.388511 26873 hybrid_clock.cc:648] HybridClock initialized: now 1781997276388456 us; error 53 us; skew 500 ppm
I20260620 23:14:36.388818 26873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:14:36 dist-test-slave-jkbq krb5kdc[26500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997276, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:14:36.392063 26873 init.cc:377] Logged in from keytab as kudu/127.10.191.195@KRBTEST.COM (short username kudu)
I20260620 23:14:36.393512 26873 webserver.cc:460] Webserver started at http://127.10.191.195:35375/ using document root <none> and password file <none>
I20260620 23:14:36.394117 26873 fs_manager.cc:359] Metadata directory not provided
I20260620 23:14:36.394214 26873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:14:36.394470 26873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:14:36.397306 26873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/data/instance:
uuid: "714cc984998f43f4b0ea18c70fa7382d"
format_stamp: "Formatted at 2026-06-20 23:14:36 on dist-test-slave-jkbq"
server_key: "2305fd1c25c65057b377af8592c2f6d4"
server_key_iv: "5a234915dde2c4e6e0084737c6e6d608"
server_key_version: "encryptionkey@0"
I20260620 23:14:36.398073 26873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance:
uuid: "714cc984998f43f4b0ea18c70fa7382d"
format_stamp: "Formatted at 2026-06-20 23:14:36 on dist-test-slave-jkbq"
server_key: "2305fd1c25c65057b377af8592c2f6d4"
server_key_iv: "5a234915dde2c4e6e0084737c6e6d608"
server_key_version: "encryptionkey@0"
I20260620 23:14:36.403218 26873 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 23:14:36.406718 26888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:36.408027 26873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:14:36.408212 26873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/wal
uuid: "714cc984998f43f4b0ea18c70fa7382d"
format_stamp: "Formatted at 2026-06-20 23:14:36 on dist-test-slave-jkbq"
server_key: "2305fd1c25c65057b377af8592c2f6d4"
server_key_iv: "5a234915dde2c4e6e0084737c6e6d608"
server_key_version: "encryptionkey@0"
I20260620 23:14:36.408427 26873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:14:36.434201 26873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:14:36.438176 26873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:14:36.438416 26873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:14:36.439414 26873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:14:36.440670 26873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:14:36.440739 26873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:36.440821 26873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:14:36.440864 26873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:36.473788 26873 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.195:33237
I20260620 23:14:36.473861 27000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.195:33237 every 8 connection(s)
I20260620 23:14:36.475101 26873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
I20260620 23:14:36.479575 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 26873
I20260620 23:14:36.479774 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance
I20260620 23:14:36.480350 11007 external_mini_cluster.cc:1428] Setting key 092fd7360fec7a7d995d85afb8e8dcfe
Jun 20 23:14:36 dist-test-slave-jkbq krb5kdc[26500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997276, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:14:36.495358 26736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:36.477522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:54203 (local address 127.10.191.254:32873)
0620 23:14:36.477696 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:36.477701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:36.478774 (+ 1073us) server_negotiation.cc:407] Connection header received
0620 23:14:36.481295 (+ 2521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:36.481301 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:36.481400 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:36.481513 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:36.483237 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:36.484146 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:36.485688 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:36.486115 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:36.486249 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:36.489662 (+ 3413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:36.489667 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:36.489670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:36.489701 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:36.492256 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:36.492977 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:36.492980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:36.492983 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:36.493108 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:36.493537 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:36.493548 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:36.493551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:36.493910 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:36.494084 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:36.494752 (+ 668us) server_negotiation.cc:299] Negotiation successful
0620 23:14:36.495008 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 23:14:36.495342 26734 heartbeater.cc:503] Master 127.10.191.254:32873 was elected leader, sending a full tablet report...
I20260620 23:14:36.496362 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:36.477893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.195:54203)
0620 23:14:36.478582 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:36.478618 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:36.480999 (+ 2381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:36.481788 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:36.481799 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:36.482459 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:14:36.483078 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:36.483097 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:36.484302 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:36.484310 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:36.485532 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:36.485549 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:36.486342 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:36.486350 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:36.486595 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:36.487294 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:14:36.487316 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:14:36.489466 (+ 2150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:14:36.492396 (+ 2930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:36.492404 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:36.492426 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:36.492837 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:36.493220 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:36.493225 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:36.493229 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:36.493408 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:36.494044 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:14:36.494053 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:14:36.494376 (+ 323us) client_negotiation.cc:769] Sending connection context
0620 23:14:36.494641 (+ 265us) client_negotiation.cc:240] Negotiation successful
0620 23:14:36.495188 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":257,"threads_started":1}
I20260620 23:14:36.497750 27001 heartbeater.cc:347] Connected to a master server at 127.10.191.254:32873
I20260620 23:14:36.498005 27001 heartbeater.cc:464] Registering TS with master...
I20260620 23:14:36.498631 27001 heartbeater.cc:511] Master 127.10.191.254:32873 requested a full tablet report, sending...
I20260620 23:14:36.499966 26545 ts_manager.cc:194] Registered new tserver with Master: 714cc984998f43f4b0ea18c70fa7382d (127.10.191.195:33237)
I20260620 23:14:36.500701 26545 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.195@KRBTEST.COM'} at 127.10.191.195:54203
I20260620 23:14:36.507772 11007 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:14:36.524780 26736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:36.512541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34288 (local address 127.10.191.254:32873)
0620 23:14:36.512788 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:36.512796 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:36.513151 (+ 355us) server_negotiation.cc:407] Connection header received
0620 23:14:36.513295 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:36.513299 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:36.513359 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:36.513436 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:36.514933 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:36.515648 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:36.516908 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:36.517306 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:36.517393 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:36.519864 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:36.519868 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:36.519871 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:36.519901 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:36.522438 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:36.523118 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:36.523122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:36.523124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:36.523246 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:36.523562 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:36.523565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:36.523567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:36.523874 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:36.524052 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:36.524285 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 23:14:36.524481 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 23:14:36.528695 26545 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34288:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:14:36.535485 26545 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:14:36.558504 27017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:36.549589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42066 (local address 127.10.191.195:33237)
0620 23:14:36.550179 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:36.550187 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:36.550795 (+ 608us) server_negotiation.cc:407] Connection header received
0620 23:14:36.551144 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:36.551153 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:36.551374 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:36.551500 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:14:36.552209 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:36.553382 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:36.556037 (+ 2655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:36.556983 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:36.557184 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:36.557367 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:36.557981 (+ 614us) server_negotiation.cc:299] Negotiation successful
0620 23:14:36.558163 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":532,"threads_started":1}
I20260620 23:14:36.558660 27014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:36.547064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44344 (local address 127.10.191.193:33111)
0620 23:14:36.549553 (+ 2489us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:36.549562 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:36.549581 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:14:36.549658 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:36.549665 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:36.549886 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:36.550399 (+ 513us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:14:36.551255 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:36.552867 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:36.555976 (+ 3109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:36.557413 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:36.557923 (+ 510us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:36.558106 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:36.558184 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 23:14:36.558335 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2332,"thread_start_us":2187,"threads_started":1}
I20260620 23:14:36.558701 27016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:36.548120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39700 (local address 127.10.191.194:37623)
0620 23:14:36.549047 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:36.549059 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:36.549506 (+ 447us) server_negotiation.cc:407] Connection header received
0620 23:14:36.549622 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:36.549630 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:36.549859 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:36.550974 (+ 1115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:14:36.551203 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:36.552868 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:36.555653 (+ 2785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:36.557128 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:36.557291 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:36.557456 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:36.558229 (+ 773us) server_negotiation.cc:299] Negotiation successful
0620 23:14:36.558389 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":548,"threads_started":1}
I20260620 23:14:36.558703 27015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:36.547856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.194:37623 (local address 127.0.0.1:39700)
0620 23:14:36.549266 (+ 1410us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:36.549286 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:36.549403 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:36.550111 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:36.550118 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:36.550166 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:14:36.550731 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:36.550750 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:36.553057 (+ 2307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:36.553064 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:36.555498 (+ 2434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:36.555511 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:36.557619 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:36.557630 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:36.557948 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:36.558005 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 23:14:36.558095 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 23:14:36.558297 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1279,"mutex_wait_us":222,"spinlock_wait_cycles":7808,"thread_start_us":1133,"threads_started":1}
I20260620 23:14:36.559381 27013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:36.546716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:33111 (local address 127.0.0.1:44344)
0620 23:14:36.547656 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:36.547713 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:36.548367 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:36.550111 (+ 1744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:36.550118 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:36.550167 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:14:36.551071 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:36.551083 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:36.553013 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:36.553018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:36.555809 (+ 2791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:36.555829 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:36.557646 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:36.557651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:36.557961 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:36.558011 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 23:14:36.558070 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 23:14:36.559008 (+ 938us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"mutex_wait_us":927,"spinlock_wait_cycles":4608,"thread_start_us":475,"threads_started":1}
I20260620 23:14:36.559458 27018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:36.549993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.195:33237 (local address 127.0.0.1:42066)
0620 23:14:36.550682 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:36.550702 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:36.550943 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:36.551735 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:36.551737 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:36.551756 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:14:36.552032 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:36.552039 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:36.553545 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:36.553549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:36.555843 (+ 2294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:36.555853 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:36.557495 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:36.557501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:36.557714 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:36.557768 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 23:14:36.558969 (+ 1201us) client_negotiation.cc:240] Negotiation successful
0620 23:14:36.559117 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":3,"thread_start_us":391,"threads_started":1}
I20260620 23:14:36.563217 26669 tablet_service.cc:1467] Processing CreateTablet for tablet 1479e13c537b47e390183823e3427b57 (DEFAULT_TABLE table=test-table [id=da6781a9c23a49488b19ecb395cf2ab1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:14:36.563241 26803 tablet_service.cc:1467] Processing CreateTablet for tablet 1479e13c537b47e390183823e3427b57 (DEFAULT_TABLE table=test-table [id=da6781a9c23a49488b19ecb395cf2ab1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:14:36.563239 26936 tablet_service.cc:1467] Processing CreateTablet for tablet 1479e13c537b47e390183823e3427b57 (DEFAULT_TABLE table=test-table [id=da6781a9c23a49488b19ecb395cf2ab1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:14:36.564481 26669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1479e13c537b47e390183823e3427b57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:14:36.564531 26803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1479e13c537b47e390183823e3427b57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:14:36.564533 26936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1479e13c537b47e390183823e3427b57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:14:36.575842 27019 tablet_bootstrap.cc:492] T 1479e13c537b47e390183823e3427b57 P c5ff4a994ed24d10ae1b3b1f8fef3789: Bootstrap starting.
I20260620 23:14:36.579893 27019 tablet_bootstrap.cc:654] T 1479e13c537b47e390183823e3427b57 P c5ff4a994ed24d10ae1b3b1f8fef3789: Neither blocks nor log segments found. Creating new log.
I20260620 23:14:36.581099 27019 log.cc:824] T 1479e13c537b47e390183823e3427b57 P c5ff4a994ed24d10ae1b3b1f8fef3789: Log is configured to *not* fsync() on all Append() calls
I20260620 23:14:36.584008 27019 tablet_bootstrap.cc:492] T 1479e13c537b47e390183823e3427b57 P c5ff4a994ed24d10ae1b3b1f8fef3789: No bootstrap required, opened a new log
I20260620 23:14:36.584280 27019 ts_tablet_manager.cc:1397] T 1479e13c537b47e390183823e3427b57 P c5ff4a994ed24d10ae1b3b1f8fef3789: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 23:14:36.587582 27021 tablet_bootstrap.cc:492] T 1479e13c537b47e390183823e3427b57 P 30bbb7d7a0764e89aaf87e5609420583: Bootstrap starting.
I20260620 23:14:36.587841 27020 tablet_bootstrap.cc:492] T 1479e13c537b47e390183823e3427b57 P 714cc984998f43f4b0ea18c70fa7382d: Bootstrap starting.
I20260620 23:14:36.590690 27020 tablet_bootstrap.cc:654] T 1479e13c537b47e390183823e3427b57 P 714cc984998f43f4b0ea18c70fa7382d: Neither blocks nor log segments found. Creating new log.
I20260620 23:14:36.591737 27020 log.cc:824] T 1479e13c537b47e390183823e3427b57 P 714cc984998f43f4b0ea18c70fa7382d: Log is configured to *not* fsync() on all Append() calls
I20260620 23:14:36.592725 27019 raft_consensus.cc:348] T 1479e13c537b47e390183823e3427b57 P c5ff4a994ed24d10ae1b3b1f8fef3789 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ff4a994ed24d10ae1b3b1f8fef3789" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 33111 } } peers { permanent_uuid: "30bbb7d7a0764e89aaf87e5609420583" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37623 } } peers { permanent_uuid: "714cc984998f43f4b0ea18c70fa7382d" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 33237 } }
I20260620 23:14:36.593173 27019 raft_consensus.cc:374] T 1479e13c537b47e390183823e3427b57 P c5ff4a994ed24d10ae1b3b1f8fef3789 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:14:36.593281 27019 raft_consensus.cc:729] T 1479e13c537b47e390183823e3427b57 P c5ff4a994ed24d10ae1b3b1f8fef3789 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ff4a994ed24d10ae1b3b1f8fef3789, State: Initialized, Role: FOLLOWER
I20260620 23:14:36.593979 27019 consensus_queue.cc:260] T 1479e13c537b47e390183823e3427b57 P c5ff4a994ed24d10ae1b3b1f8fef3789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ff4a994ed24d10ae1b3b1f8fef3789" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 33111 } } peers { permanent_uuid: "30bbb7d7a0764e89aaf87e5609420583" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37623 } } peers { permanent_uuid: "714cc984998f43f4b0ea18c70fa7382d" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 33237 } }
I20260620 23:14:36.594277 27021 tablet_bootstrap.cc:654] T 1479e13c537b47e390183823e3427b57 P 30bbb7d7a0764e89aaf87e5609420583: Neither blocks nor log segments found. Creating new log.
I20260620 23:14:36.594547 27020 tablet_bootstrap.cc:492] T 1479e13c537b47e390183823e3427b57 P 714cc984998f43f4b0ea18c70fa7382d: No bootstrap required, opened a new log
I20260620 23:14:36.594871 27020 ts_tablet_manager.cc:1397] T 1479e13c537b47e390183823e3427b57 P 714cc984998f43f4b0ea18c70fa7382d: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20260620 23:14:36.595589 27021 log.cc:824] T 1479e13c537b47e390183823e3427b57 P 30bbb7d7a0764e89aaf87e5609420583: Log is configured to *not* fsync() on all Append() calls
I20260620 23:14:36.595865 27019 ts_tablet_manager.cc:1428] T 1479e13c537b47e390183823e3427b57 P c5ff4a994ed24d10ae1b3b1f8fef3789: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260620 23:14:36.598809 27021 tablet_bootstrap.cc:492] T 1479e13c537b47e390183823e3427b57 P 30bbb7d7a0764e89aaf87e5609420583: No bootstrap required, opened a new log
I20260620 23:14:36.599138 27021 ts_tablet_manager.cc:1397] T 1479e13c537b47e390183823e3427b57 P 30bbb7d7a0764e89aaf87e5609420583: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260620 23:14:36.605212 27020 raft_consensus.cc:348] T 1479e13c537b47e390183823e3427b57 P 714cc984998f43f4b0ea18c70fa7382d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ff4a994ed24d10ae1b3b1f8fef3789" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 33111 } } peers { permanent_uuid: "30bbb7d7a0764e89aaf87e5609420583" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37623 } } peers { permanent_uuid: "714cc984998f43f4b0ea18c70fa7382d" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 33237 } }
I20260620 23:14:36.605659 27020 raft_consensus.cc:374] T 1479e13c537b47e390183823e3427b57 P 714cc984998f43f4b0ea18c70fa7382d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:14:36.605760 27020 raft_consensus.cc:729] T 1479e13c537b47e390183823e3427b57 P 714cc984998f43f4b0ea18c70fa7382d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 714cc984998f43f4b0ea18c70fa7382d, State: Initialized, Role: FOLLOWER
I20260620 23:14:36.606372 27020 consensus_queue.cc:260] T 1479e13c537b47e390183823e3427b57 P 714cc984998f43f4b0ea18c70fa7382d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ff4a994ed24d10ae1b3b1f8fef3789" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 33111 } } peers { permanent_uuid: "30bbb7d7a0764e89aaf87e5609420583" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37623 } } peers { permanent_uuid: "714cc984998f43f4b0ea18c70fa7382d" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 33237 } }
I20260620 23:14:36.607471 27001 heartbeater.cc:503] Master 127.10.191.254:32873 was elected leader, sending a full tablet report...
I20260620 23:14:36.608156 27020 ts_tablet_manager.cc:1428] T 1479e13c537b47e390183823e3427b57 P 714cc984998f43f4b0ea18c70fa7382d: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260620 23:14:36.609549 27021 raft_consensus.cc:348] T 1479e13c537b47e390183823e3427b57 P 30bbb7d7a0764e89aaf87e5609420583 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ff4a994ed24d10ae1b3b1f8fef3789" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 33111 } } peers { permanent_uuid: "30bbb7d7a0764e89aaf87e5609420583" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37623 } } peers { permanent_uuid: "714cc984998f43f4b0ea18c70fa7382d" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 33237 } }
I20260620 23:14:36.609936 27021 raft_consensus.cc:374] T 1479e13c537b47e390183823e3427b57 P 30bbb7d7a0764e89aaf87e5609420583 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:14:36.610037 27021 raft_consensus.cc:729] T 1479e13c537b47e390183823e3427b57 P 30bbb7d7a0764e89aaf87e5609420583 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30bbb7d7a0764e89aaf87e5609420583, State: Initialized, Role: FOLLOWER
I20260620 23:14:36.610580 27021 consensus_queue.cc:260] T 1479e13c537b47e390183823e3427b57 P 30bbb7d7a0764e89aaf87e5609420583 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ff4a994ed24d10ae1b3b1f8fef3789" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 33111 } } peers { permanent_uuid: "30bbb7d7a0764e89aaf87e5609420583" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37623 } } peers { permanent_uuid: "714cc984998f43f4b0ea18c70fa7382d" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 33237 } }
I20260620 23:14:36.611632 26868 heartbeater.cc:503] Master 127.10.191.254:32873 was elected leader, sending a full tablet report...
I20260620 23:14:36.612214 27021 ts_tablet_manager.cc:1428] T 1479e13c537b47e390183823e3427b57 P 30bbb7d7a0764e89aaf87e5609420583: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
W20260620 23:14:36.713986 26735 tablet.cc:2369] T 1479e13c537b47e390183823e3427b57 P c5ff4a994ed24d10ae1b3b1f8fef3789: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:14:36.716830 26869 tablet.cc:2369] T 1479e13c537b47e390183823e3427b57 P 30bbb7d7a0764e89aaf87e5609420583: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:14:36.726682 27002 tablet.cc:2369] T 1479e13c537b47e390183823e3427b57 P 714cc984998f43f4b0ea18c70fa7382d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:14:36.786011 27027 raft_consensus.cc:482] T 1479e13c537b47e390183823e3427b57 P 30bbb7d7a0764e89aaf87e5609420583 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:14:36.786377 27027 raft_consensus.cc:504] T 1479e13c537b47e390183823e3427b57 P 30bbb7d7a0764e89aaf87e5609420583 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ff4a994ed24d10ae1b3b1f8fef3789" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 33111 } } peers { permanent_uuid: "30bbb7d7a0764e89aaf87e5609420583" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37623 } } peers { permanent_uuid: "714cc984998f43f4b0ea18c70fa7382d" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 33237 } }
I20260620 23:14:36.788164 27027 leader_election.cc:288] T 1479e13c537b47e390183823e3427b57 P 30bbb7d7a0764e89aaf87e5609420583 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ff4a994ed24d10ae1b3b1f8fef3789 (127.10.191.193:33111), 714cc984998f43f4b0ea18c70fa7382d (127.10.191.195:33237)
I20260620 23:14:36.795812 27014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:36.788498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:50803 (local address 127.10.191.193:33111)
0620 23:14:36.788695 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:36.788699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:36.789322 (+ 623us) server_negotiation.cc:407] Connection header received
0620 23:14:36.789529 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:36.789535 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:36.789619 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:36.789720 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:14:36.790388 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:36.791276 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:36.793703 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:36.794964 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:36.795131 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:36.795210 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:36.795477 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 23:14:36.795597 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 23:14:36.795974 27028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:36.788487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:33111 (local address 127.10.191.194:50803)
0620 23:14:36.789200 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:36.789219 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:36.789352 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:36.789800 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:36.789805 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:36.789833 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:14:36.790207 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:36.790215 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:36.791428 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:36.791435 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:36.793528 (+ 2093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:36.793543 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:36.795174 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:36.795178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:36.795292 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:36.795316 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 23:14:36.795443 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 23:14:36.795588 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":387,"threads_started":1}
I20260620 23:14:36.796288 27029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:36.789300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.195:33237 (local address 127.10.191.194:38223)
0620 23:14:36.789943 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:36.789957 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:36.790100 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:36.790437 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:36.790441 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:36.790463 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:14:36.790772 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:36.790780 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:36.791998 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:36.792006 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:36.793903 (+ 1897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:36.793912 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:36.795475 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:36.795481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:36.795647 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:36.795682 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 23:14:36.795774 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 23:14:36.795883 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":342,"threads_started":1}
I20260620 23:14:36.796744 27017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:36.789492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:38223 (local address 127.10.191.195:33237)
0620 23:14:36.789695 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:36.789703 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:36.790050 (+ 347us) server_negotiation.cc:407] Connection header received
0620 23:14:36.790246 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:36.790267 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:36.790325 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:36.790423 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:14:36.791014 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:36.791867 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:36.794035 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:36.795274 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:36.795394 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:36.795486 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:36.795826 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 23:14:36.796552 (+ 726us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 23:14:36.797024 26689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1479e13c537b47e390183823e3427b57" candidate_uuid: "30bbb7d7a0764e89aaf87e5609420583" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ff4a994ed24d10ae1b3b1f8fef3789" is_pre_election: true
I20260620 23:14:36.797533 26689 raft_consensus.cc:2454] T 1479e13c537b47e390183823e3427b57 P c5ff4a994ed24d10ae1b3b1f8fef3789 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30bbb7d7a0764e89aaf87e5609420583 in term 0.
I20260620 23:14:36.797542 26956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1479e13c537b47e390183823e3427b57" candidate_uuid: "30bbb7d7a0764e89aaf87e5609420583" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "714cc984998f43f4b0ea18c70fa7382d" is_pre_election: true
I20260620 23:14:36.798082 26956 raft_consensus.cc:2454] T 1479e13c537b47e390183823e3427b57 P 714cc984998f43f4b0ea18c70fa7382d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30bbb7d7a0764e89aaf87e5609420583 in term 0.
I20260620 23:14:36.798240 26757 leader_election.cc:302] T 1479e13c537b47e390183823e3427b57 P 30bbb7d7a0764e89aaf87e5609420583 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30bbb7d7a0764e89aaf87e5609420583, c5ff4a994ed24d10ae1b3b1f8fef3789; no voters:
I20260620 23:14:36.798677 27027 raft_consensus.cc:2781] T 1479e13c537b47e390183823e3427b57 P 30bbb7d7a0764e89aaf87e5609420583 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:14:36.798854 27027 raft_consensus.cc:482] T 1479e13c537b47e390183823e3427b57 P 30bbb7d7a0764e89aaf87e5609420583 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:14:36.798992 27027 raft_consensus.cc:3037] T 1479e13c537b47e390183823e3427b57 P 30bbb7d7a0764e89aaf87e5609420583 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:14:36.801118 27027 raft_consensus.cc:504] T 1479e13c537b47e390183823e3427b57 P 30bbb7d7a0764e89aaf87e5609420583 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ff4a994ed24d10ae1b3b1f8fef3789" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 33111 } } peers { permanent_uuid: "30bbb7d7a0764e89aaf87e5609420583" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37623 } } peers { permanent_uuid: "714cc984998f43f4b0ea18c70fa7382d" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 33237 } }
I20260620 23:14:36.801756 27027 leader_election.cc:288] T 1479e13c537b47e390183823e3427b57 P 30bbb7d7a0764e89aaf87e5609420583 [CANDIDATE]: Term 1 election: Requested vote from peers c5ff4a994ed24d10ae1b3b1f8fef3789 (127.10.191.193:33111), 714cc984998f43f4b0ea18c70fa7382d (127.10.191.195:33237)
I20260620 23:14:36.802168 26689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1479e13c537b47e390183823e3427b57" candidate_uuid: "30bbb7d7a0764e89aaf87e5609420583" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ff4a994ed24d10ae1b3b1f8fef3789"
I20260620 23:14:36.802291 26956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1479e13c537b47e390183823e3427b57" candidate_uuid: "30bbb7d7a0764e89aaf87e5609420583" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "714cc984998f43f4b0ea18c70fa7382d"
I20260620 23:14:36.802399 26689 raft_consensus.cc:3037] T 1479e13c537b47e390183823e3427b57 P c5ff4a994ed24d10ae1b3b1f8fef3789 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:14:36.802492 26956 raft_consensus.cc:3037] T 1479e13c537b47e390183823e3427b57 P 714cc984998f43f4b0ea18c70fa7382d [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:14:36.804708 26689 raft_consensus.cc:2454] T 1479e13c537b47e390183823e3427b57 P c5ff4a994ed24d10ae1b3b1f8fef3789 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30bbb7d7a0764e89aaf87e5609420583 in term 1.
I20260620 23:14:36.805096 26757 leader_election.cc:302] T 1479e13c537b47e390183823e3427b57 P 30bbb7d7a0764e89aaf87e5609420583 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30bbb7d7a0764e89aaf87e5609420583, c5ff4a994ed24d10ae1b3b1f8fef3789; no voters:
I20260620 23:14:36.805397 27027 raft_consensus.cc:2781] T 1479e13c537b47e390183823e3427b57 P 30bbb7d7a0764e89aaf87e5609420583 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:14:36.805608 26956 raft_consensus.cc:2454] T 1479e13c537b47e390183823e3427b57 P 714cc984998f43f4b0ea18c70fa7382d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30bbb7d7a0764e89aaf87e5609420583 in term 1.
I20260620 23:14:36.808249 27027 raft_consensus.cc:686] T 1479e13c537b47e390183823e3427b57 P 30bbb7d7a0764e89aaf87e5609420583 [term 1 LEADER]: Becoming Leader. State: Replica: 30bbb7d7a0764e89aaf87e5609420583, State: Running, Role: LEADER
I20260620 23:14:36.808843 27027 consensus_queue.cc:237] T 1479e13c537b47e390183823e3427b57 P 30bbb7d7a0764e89aaf87e5609420583 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ff4a994ed24d10ae1b3b1f8fef3789" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 33111 } } peers { permanent_uuid: "30bbb7d7a0764e89aaf87e5609420583" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37623 } } peers { permanent_uuid: "714cc984998f43f4b0ea18c70fa7382d" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 33237 } }
I20260620 23:14:36.813933 26544 catalog_manager.cc:5495] T 1479e13c537b47e390183823e3427b57 P 30bbb7d7a0764e89aaf87e5609420583 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30bbb7d7a0764e89aaf87e5609420583 (127.10.191.194). New cstate: current_term: 1 leader_uuid: "30bbb7d7a0764e89aaf87e5609420583" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ff4a994ed24d10ae1b3b1f8fef3789" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 33111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30bbb7d7a0764e89aaf87e5609420583" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "714cc984998f43f4b0ea18c70fa7382d" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 33237 } health_report { overall_health: UNKNOWN } } }
I20260620 23:14:36.856566 27016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:36.850746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39712 (local address 127.10.191.194:37623)
0620 23:14:36.850985 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:36.850991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:36.851096 (+ 105us) server_negotiation.cc:407] Connection header received
0620 23:14:36.851348 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:36.851352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:36.851447 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:36.851604 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:14:36.852009 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:36.852764 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:36.854263 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:36.854681 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:36.854820 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:36.855328 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:14:36.855945 (+ 617us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:14:36.856103 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:36.856212 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 23:14:36.856293 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 23:14:36.864480 26689 raft_consensus.cc:1261] T 1479e13c537b47e390183823e3427b57 P c5ff4a994ed24d10ae1b3b1f8fef3789 [term 1 FOLLOWER]: Refusing update from remote peer 30bbb7d7a0764e89aaf87e5609420583: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:14:36.864471 26956 raft_consensus.cc:1261] T 1479e13c537b47e390183823e3427b57 P 714cc984998f43f4b0ea18c70fa7382d [term 1 FOLLOWER]: Refusing update from remote peer 30bbb7d7a0764e89aaf87e5609420583: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:14:36.865450 27030 consensus_queue.cc:1035] T 1479e13c537b47e390183823e3427b57 P 30bbb7d7a0764e89aaf87e5609420583 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ff4a994ed24d10ae1b3b1f8fef3789" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 33111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:14:36.865875 27027 consensus_queue.cc:1035] T 1479e13c537b47e390183823e3427b57 P 30bbb7d7a0764e89aaf87e5609420583 [LEADER]: Connected to new peer: Peer: permanent_uuid: "714cc984998f43f4b0ea18c70fa7382d" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:14:36.877548 27033 mvcc.cc:205] Tried to move back new op lower bound from 7299060846027202560 to 7299060845813284864. Current Snapshot: MvccSnapshot[applied={T|T < 7299060846027202560}]
I20260620 23:14:36.878743 27034 mvcc.cc:205] Tried to move back new op lower bound from 7299060846027202560 to 7299060845813284864. Current Snapshot: MvccSnapshot[applied={T|T < 7299060846027202560}]
I20260620 23:14:36.892212 26545 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:14:36.892478 26545 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:14:36.896315 26545 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3: Sending DeleteTablet for 3 replicas of tablet 1479e13c537b47e390183823e3427b57
I20260620 23:14:36.897382 26669 tablet_service.cc:1514] Processing DeleteTablet for tablet 1479e13c537b47e390183823e3427b57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:14:36 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:44344
I20260620 23:14:36.897571 26936 tablet_service.cc:1514] Processing DeleteTablet for tablet 1479e13c537b47e390183823e3427b57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:14:36 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:42066
I20260620 23:14:36.898341 27043 tablet_replica.cc:330] T 1479e13c537b47e390183823e3427b57 P c5ff4a994ed24d10ae1b3b1f8fef3789: stopping tablet replica
I20260620 23:14:36.898258 26803 tablet_service.cc:1514] Processing DeleteTablet for tablet 1479e13c537b47e390183823e3427b57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:14:36 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:39700
I20260620 23:14:36.898625 27044 tablet_replica.cc:330] T 1479e13c537b47e390183823e3427b57 P 714cc984998f43f4b0ea18c70fa7382d: stopping tablet replica
I20260620 23:14:36.898787 27043 raft_consensus.cc:2229] T 1479e13c537b47e390183823e3427b57 P c5ff4a994ed24d10ae1b3b1f8fef3789 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:14:36.899127 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 26606
I20260620 23:14:36.899165 27044 raft_consensus.cc:2229] T 1479e13c537b47e390183823e3427b57 P 714cc984998f43f4b0ea18c70fa7382d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:14:36.899381 27043 raft_consensus.cc:2258] T 1479e13c537b47e390183823e3427b57 P c5ff4a994ed24d10ae1b3b1f8fef3789 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:14:36.899791 27044 raft_consensus.cc:2258] T 1479e13c537b47e390183823e3427b57 P 714cc984998f43f4b0ea18c70fa7382d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:14:36.901678 27043 ts_tablet_manager.cc:1905] T 1479e13c537b47e390183823e3427b57 P c5ff4a994ed24d10ae1b3b1f8fef3789: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:14:36.902038 27044 ts_tablet_manager.cc:1905] T 1479e13c537b47e390183823e3427b57 P 714cc984998f43f4b0ea18c70fa7382d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:14:36.905797 27045 tablet_replica.cc:330] T 1479e13c537b47e390183823e3427b57 P 30bbb7d7a0764e89aaf87e5609420583: stopping tablet replica
I20260620 23:14:36.906226 27045 raft_consensus.cc:2229] T 1479e13c537b47e390183823e3427b57 P 30bbb7d7a0764e89aaf87e5609420583 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:14:36.907393 27045 raft_consensus.cc:2258] T 1479e13c537b47e390183823e3427b57 P 30bbb7d7a0764e89aaf87e5609420583 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:14:36.909683 27043 ts_tablet_manager.cc:1918] T 1479e13c537b47e390183823e3427b57 P c5ff4a994ed24d10ae1b3b1f8fef3789: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:14:36.909904 27043 log.cc:1196] T 1479e13c537b47e390183823e3427b57 P c5ff4a994ed24d10ae1b3b1f8fef3789: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/wal/wals/1479e13c537b47e390183823e3427b57
I20260620 23:14:36.910485 27043 ts_tablet_manager.cc:1939] T 1479e13c537b47e390183823e3427b57 P c5ff4a994ed24d10ae1b3b1f8fef3789: Deleting consensus metadata
I20260620 23:14:36.911180 27045 ts_tablet_manager.cc:1905] T 1479e13c537b47e390183823e3427b57 P 30bbb7d7a0764e89aaf87e5609420583: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:14:36.912499 27044 ts_tablet_manager.cc:1918] T 1479e13c537b47e390183823e3427b57 P 714cc984998f43f4b0ea18c70fa7382d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:14:36.912664 27044 log.cc:1196] T 1479e13c537b47e390183823e3427b57 P 714cc984998f43f4b0ea18c70fa7382d: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/wal/wals/1479e13c537b47e390183823e3427b57
I20260620 23:14:36.912781 26530 catalog_manager.cc:4841] TS c5ff4a994ed24d10ae1b3b1f8fef3789 (127.10.191.193:33111): tablet 1479e13c537b47e390183823e3427b57 (table test-table [id=da6781a9c23a49488b19ecb395cf2ab1]) successfully deleted
I20260620 23:14:36.913182 27044 ts_tablet_manager.cc:1939] T 1479e13c537b47e390183823e3427b57 P 714cc984998f43f4b0ea18c70fa7382d: Deleting consensus metadata
I20260620 23:14:36.914902 26530 catalog_manager.cc:4841] TS 714cc984998f43f4b0ea18c70fa7382d (127.10.191.195:33237): tablet 1479e13c537b47e390183823e3427b57 (table test-table [id=da6781a9c23a49488b19ecb395cf2ab1]) successfully deleted
I20260620 23:14:36.918659 27045 ts_tablet_manager.cc:1918] T 1479e13c537b47e390183823e3427b57 P 30bbb7d7a0764e89aaf87e5609420583: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:14:36.918836 27045 log.cc:1196] T 1479e13c537b47e390183823e3427b57 P 30bbb7d7a0764e89aaf87e5609420583: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/wal/wals/1479e13c537b47e390183823e3427b57
Jun 20 23:14:36 dist-test-slave-jkbq krb5kdc[26500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.193@KRBTEST.COM
I20260620 23:14:36.919469 27045 ts_tablet_manager.cc:1939] T 1479e13c537b47e390183823e3427b57 P 30bbb7d7a0764e89aaf87e5609420583: Deleting consensus metadata
I20260620 23:14:36.921237 26530 catalog_manager.cc:4841] TS 30bbb7d7a0764e89aaf87e5609420583 (127.10.191.194:37623): tablet 1479e13c537b47e390183823e3427b57 (table test-table [id=da6781a9c23a49488b19ecb395cf2ab1]) successfully deleted
I20260620 23:14:36.927649 27014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:36.900238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44350 (local address 127.10.191.193:33111)
0620 23:14:36.900603 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:36.900608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:36.907024 (+ 6416us) server_negotiation.cc:407] Connection header received
0620 23:14:36.907309 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:36.907315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:36.907410 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:36.907490 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:36.909908 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:36.911117 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:36.913090 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:36.913733 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:36.913858 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:36.920703 (+ 6845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:36.920719 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:36.920738 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:36.920782 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:36.924634 (+ 3852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:36.925369 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:36.925379 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:36.925383 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:36.925563 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:36.925941 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:36.925946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:36.925949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:36.926396 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:36.926711 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:36.927038 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 23:14:36.927265 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242}
I20260620 23:14:36.928297 26729 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:14:37.021044 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 26606
I20260620 23:14:37.038537 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 26740
Jun 20 23:14:37 dist-test-slave-jkbq krb5kdc[26500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.194@KRBTEST.COM
I20260620 23:14:37.056946 27016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:37.039688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39718 (local address 127.10.191.194:37623)
0620 23:14:37.039900 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:37.039909 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:37.039937 (+ 28us) server_negotiation.cc:407] Connection header received
0620 23:14:37.040007 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:37.040013 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:37.040095 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:37.040210 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:37.041598 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:37.042843 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:37.044327 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:37.045022 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:37.045156 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:37.049513 (+ 4357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:37.049533 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:37.049555 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:37.049614 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:37.053663 (+ 4049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:37.054387 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:37.054396 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:37.054400 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:37.054586 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:37.055077 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:37.055082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:37.055086 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:37.055565 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:37.055911 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:37.056285 (+ 374us) server_negotiation.cc:299] Negotiation successful
0620 23:14:37.056519 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 23:14:37.057777 26863 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:14:37.172703 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 26740
I20260620 23:14:37.191964 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 26873
Jun 20 23:14:37 dist-test-slave-jkbq krb5kdc[26500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.195@KRBTEST.COM
I20260620 23:14:37.206753 27017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:37.192687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42080 (local address 127.10.191.195:33237)
0620 23:14:37.192876 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:37.192880 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:37.192898 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:14:37.193045 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:37.193049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:37.193127 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:37.193213 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:37.194622 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:37.195437 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:37.196728 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:37.197100 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:37.197193 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:37.201498 (+ 4305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:37.201514 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:37.201530 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:37.201567 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:37.204267 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:37.204903 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:37.204910 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:37.204913 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:37.205054 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:37.205485 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:37.205488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:37.205490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:37.205828 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:37.206049 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:37.206322 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 23:14:37.206497 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 23:14:37.207500 26996 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:14:37.295910 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 26873
I20260620 23:14:37.312604 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 26513
I20260620 23:14:37.313829 26575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:14:37.425482 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 26513
I20260620 23:14:57.447211 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:32873
--webserver_interface=127.10.191.254
--webserver_port=34225
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:33627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:32873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:14:57.766484 27051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:14:57.766765 27051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:14:57.766855 27051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:14:57.774194 27051 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 23:14:57.774308 27051 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 23:14:57.774348 27051 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 23:14:57.774394 27051 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:14:57.774451 27051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:14:57.774518 27051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:14:57.774564 27051 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:14:57.774611 27051 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:14:57.784453 27051 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:33627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:32873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:32873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=34225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:14:57.785663 27051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:14:57.787009 27051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:14:57.809993 27057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:57.810025 27058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:57.809902 27060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:14:57.810526 27051 server_base.cc:1034] running on GCE node
I20260620 23:14:57.811625 27051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:14:57.813211 27051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:14:57.814476 27051 hybrid_clock.cc:648] HybridClock initialized: now 1781997297814418 us; error 37 us; skew 500 ppm
I20260620 23:14:57.814975 27051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:14:57 dist-test-slave-jkbq krb5kdc[26500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997297, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:14:57.818665 27051 init.cc:377] Logged in from keytab as kudu/127.10.191.254@KRBTEST.COM (short username kudu)
I20260620 23:14:57.820392 27051 webserver.cc:460] Webserver started at http://127.10.191.254:34225/ using document root <none> and password file <none>
I20260620 23:14:57.821143 27051 fs_manager.cc:359] Metadata directory not provided
I20260620 23:14:57.821251 27051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:14:57.827822 27051 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 23:14:57.830755 27067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:57.832226 27051 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:14:57.832376 27051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "d6d9ed6d31094bb29fa59de15a5924a3"
format_stamp: "Formatted at 2026-06-20 23:14:34 on dist-test-slave-jkbq"
server_key: "2cf0710873a8459633bae0f7d6613168"
server_key_iv: "655b8f7dd28bd835c94f275f04379e4f"
server_key_version: "encryptionkey@0"
I20260620 23:14:57.833429 27051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:14:57.858134 27051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:14:57.862622 27051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:14:57.862946 27051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:14:57.881804 27051 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:32873
I20260620 23:14:57.881829 27118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:32873 every 8 connection(s)
I20260620 23:14:57.883407 27051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:14:57.889716 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 27051
I20260620 23:14:57.890342 27119 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:14:57.890623 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.193:33111
--local_ip_for_outbound_sockets=127.10.191.193
--tserver_master_addrs=127.10.191.254:32873
--webserver_port=45075
--webserver_interface=127.10.191.193
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:33627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:14:57.893788 27119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3: Bootstrap starting.
I20260620 23:14:57.920780 27119 log.cc:824] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3: Log is configured to *not* fsync() on all Append() calls
I20260620 23:14:57.936327 27119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:14:57.937041 27119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3: Bootstrap complete.
I20260620 23:14:57.951128 27119 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d9ed6d31094bb29fa59de15a5924a3" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 32873 } }
I20260620 23:14:57.952342 27119 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6d9ed6d31094bb29fa59de15a5924a3, State: Initialized, Role: FOLLOWER
I20260620 23:14:57.953081 27119 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d9ed6d31094bb29fa59de15a5924a3" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 32873 } }
I20260620 23:14:57.953372 27119 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:14:57.953503 27119 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:14:57.953687 27119 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:14:57.958005 27119 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d9ed6d31094bb29fa59de15a5924a3" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 32873 } }
I20260620 23:14:57.958544 27119 leader_election.cc:302] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6d9ed6d31094bb29fa59de15a5924a3; no voters:
I20260620 23:14:57.959275 27119 leader_election.cc:288] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 23:14:57.959409 27123 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 23:14:57.960655 27123 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [term 2 LEADER]: Becoming Leader. State: Replica: d6d9ed6d31094bb29fa59de15a5924a3, State: Running, Role: LEADER
I20260620 23:14:57.961212 27123 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d9ed6d31094bb29fa59de15a5924a3" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 32873 } }
I20260620 23:14:57.961772 27119 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:14:57.963716 27125 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6d9ed6d31094bb29fa59de15a5924a3. Latest consensus state: current_term: 2 leader_uuid: "d6d9ed6d31094bb29fa59de15a5924a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d9ed6d31094bb29fa59de15a5924a3" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 32873 } } }
I20260620 23:14:57.964094 27125 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [sys.catalog]: This master's current role is: LEADER
I20260620 23:14:57.964605 27124 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d6d9ed6d31094bb29fa59de15a5924a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d9ed6d31094bb29fa59de15a5924a3" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 32873 } } }
I20260620 23:14:57.964943 27124 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [sys.catalog]: This master's current role is: LEADER
I20260620 23:14:57.971199 27129 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:14:57.982280 27129 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:14:57.984150 27129 catalog_manager.cc:1259] Loaded cluster ID: a365fdc97ce44b7aaa187d49cea26635
I20260620 23:14:57.984266 27129 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:14:57.989429 27129 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:14:57.991909 27129 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3: Loaded TSK: 0
I20260620 23:14:58.042409 27129 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3: Generated new TSK 1
I20260620 23:14:58.042815 27129 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 23:14:58.265950 27121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:14:58.266227 27121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:14:58.266373 27121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:14:58.273881 27121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:14:58.274034 27121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:14:58.274204 27121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.193
I20260620 23:14:58.287106 27121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:33627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.193:33111
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.191.193
--webserver_port=45075
--tserver_master_addrs=127.10.191.254:32873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.193
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:14:58.288934 27121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:14:58.290408 27121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:14:58.304337 27147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:58.304307 27149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:14:58.305294 27121 server_base.cc:1034] running on GCE node
W20260620 23:14:58.307509 27146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:14:58.308254 27121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:14:58.309278 27121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:14:58.310508 27121 hybrid_clock.cc:648] HybridClock initialized: now 1781997298310431 us; error 66 us; skew 500 ppm
I20260620 23:14:58.310999 27121 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:14:58 dist-test-slave-jkbq krb5kdc[26500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997298, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:14:58.314400 27121 init.cc:377] Logged in from keytab as kudu/127.10.191.193@KRBTEST.COM (short username kudu)
I20260620 23:14:58.316075 27121 webserver.cc:460] Webserver started at http://127.10.191.193:45075/ using document root <none> and password file <none>
I20260620 23:14:58.316890 27121 fs_manager.cc:359] Metadata directory not provided
I20260620 23:14:58.317025 27121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:14:58.323841 27121 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 23:14:58.327204 27156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:58.328890 27121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:14:58.329062 27121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/wal
uuid: "c5ff4a994ed24d10ae1b3b1f8fef3789"
format_stamp: "Formatted at 2026-06-20 23:14:35 on dist-test-slave-jkbq"
server_key: "eb9a2a15ac4996c235cb9c64bf327582"
server_key_iv: "85a36910358a037b75d27b2804e55630"
server_key_version: "encryptionkey@0"
I20260620 23:14:58.330433 27121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:14:58.356683 27121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:14:58.361928 27121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:14:58.362246 27121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:14:58.363468 27121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:14:58.364979 27121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:14:58.365090 27121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:58.365192 27121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:14:58.365263 27121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:58.399569 27121 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.193:33111
I20260620 23:14:58.399618 27268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.193:33111 every 8 connection(s)
I20260620 23:14:58.400995 27121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
I20260620 23:14:58.409229 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 27121
I20260620 23:14:58.409912 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.194:37623
--local_ip_for_outbound_sockets=127.10.191.194
--tserver_master_addrs=127.10.191.254:32873
--webserver_port=40087
--webserver_interface=127.10.191.194
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:33627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:14:58 dist-test-slave-jkbq krb5kdc[26500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997298, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:14:58.426103 27271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:58.403962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:33231 (local address 127.10.191.254:32873)
0620 23:14:58.404539 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:58.404551 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:58.405199 (+ 648us) server_negotiation.cc:407] Connection header received
0620 23:14:58.407326 (+ 2127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:58.407339 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:58.407651 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:58.407938 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:58.409322 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:58.410823 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:58.413355 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:58.413990 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:58.414117 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:58.418648 (+ 4531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:58.418656 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:58.418669 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:58.418705 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:58.421676 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:58.422637 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:58.422646 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:58.422654 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:58.422825 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:58.423348 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:58.423351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:58.423353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:58.423809 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:58.423959 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:58.424879 (+ 920us) server_negotiation.cc:299] Negotiation successful
0620 23:14:58.425193 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":271,"threads_started":1}
I20260620 23:14:58.426573 27272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:58.404194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.193:33231)
0620 23:14:58.405031 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:58.405076 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:58.406776 (+ 1700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:58.408082 (+ 1306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:58.408089 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:58.408585 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:14:58.409088 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:58.409104 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:58.411106 (+ 2002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:58.411115 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:58.413207 (+ 2092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:58.413226 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:58.414249 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:58.414260 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:58.414607 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:58.415359 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:14:58.415397 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:14:58.418363 (+ 2966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:14:58.421835 (+ 3472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:58.421846 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:58.421864 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:58.422500 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:58.422997 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:58.423005 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:58.423011 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:58.423242 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:58.423963 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:14:58.423974 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:14:58.424413 (+ 439us) client_negotiation.cc:769] Sending connection context
0620 23:14:58.424724 (+ 311us) client_negotiation.cc:240] Negotiation successful
0620 23:14:58.425287 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":340,"threads_started":1}
I20260620 23:14:58.429560 27269 heartbeater.cc:347] Connected to a master server at 127.10.191.254:32873
I20260620 23:14:58.429872 27269 heartbeater.cc:464] Registering TS with master...
I20260620 23:14:58.430605 27269 heartbeater.cc:511] Master 127.10.191.254:32873 requested a full tablet report, sending...
I20260620 23:14:58.433112 27084 ts_manager.cc:194] Registered new tserver with Master: c5ff4a994ed24d10ae1b3b1f8fef3789 (127.10.191.193:33111)
I20260620 23:14:58.435417 27084 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.193@KRBTEST.COM'} at 127.10.191.193:33231
W20260620 23:14:58.731109 27273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:14:58.731375 27273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:14:58.731468 27273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:14:58.738765 27273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:14:58.738886 27273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:14:58.739045 27273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.194
I20260620 23:14:58.748494 27273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:33627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.194:37623
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.191.194
--webserver_port=40087
--tserver_master_addrs=127.10.191.254:32873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.194
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:14:58.749660 27273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:14:58.750962 27273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:14:58.763898 27279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:58.763984 27281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:58.764312 27278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:14:58.765103 27273 server_base.cc:1034] running on GCE node
I20260620 23:14:58.765779 27273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:14:58.766700 27273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:14:58.767928 27273 hybrid_clock.cc:648] HybridClock initialized: now 1781997298767873 us; error 50 us; skew 500 ppm
I20260620 23:14:58.768240 27273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:14:58 dist-test-slave-jkbq krb5kdc[26500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997298, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:14:58.771306 27273 init.cc:377] Logged in from keytab as kudu/127.10.191.194@KRBTEST.COM (short username kudu)
I20260620 23:14:58.772723 27273 webserver.cc:460] Webserver started at http://127.10.191.194:40087/ using document root <none> and password file <none>
I20260620 23:14:58.773412 27273 fs_manager.cc:359] Metadata directory not provided
I20260620 23:14:58.773567 27273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:14:58.778707 27273 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260620 23:14:58.781343 27288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:58.782684 27273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:14:58.782857 27273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/wal
uuid: "30bbb7d7a0764e89aaf87e5609420583"
format_stamp: "Formatted at 2026-06-20 23:14:35 on dist-test-slave-jkbq"
server_key: "283a04a8e4f55dee6a1fe0c77108acf8"
server_key_iv: "931086b80c320db55475310070aa1715"
server_key_version: "encryptionkey@0"
I20260620 23:14:58.783870 27273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:14:58.818426 27273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:14:58.822808 27273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:14:58.823173 27273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:14:58.824126 27273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:14:58.825407 27273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:14:58.825498 27273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:58.825575 27273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:14:58.825628 27273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:58.859035 27273 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.194:37623
I20260620 23:14:58.859076 27400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.194:37623 every 8 connection(s)
I20260620 23:14:58.860486 27273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
I20260620 23:14:58.866513 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 27273
I20260620 23:14:58.867350 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.195:33237
--local_ip_for_outbound_sockets=127.10.191.195
--tserver_master_addrs=127.10.191.254:32873
--webserver_port=35375
--webserver_interface=127.10.191.195
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:33627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:14:58 dist-test-slave-jkbq krb5kdc[26500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997298, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:14:58.882130 27271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:58.863311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:44311 (local address 127.10.191.254:32873)
0620 23:14:58.863533 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:58.863540 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:58.864817 (+ 1277us) server_negotiation.cc:407] Connection header received
0620 23:14:58.866656 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:58.866663 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:58.866750 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:58.866887 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:58.868506 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:58.869659 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:58.871375 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:58.872181 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:58.872332 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:58.876235 (+ 3903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:58.876240 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:58.876243 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:58.876291 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:58.878734 (+ 2443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:58.879637 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:58.879641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:58.879643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:58.879760 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:58.880228 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:58.880232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:58.880233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:58.880492 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:58.880604 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:58.881527 (+ 923us) server_negotiation.cc:299] Negotiation successful
0620 23:14:58.881777 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 23:14:58.883422 27403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:58.863908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.194:44311)
0620 23:14:58.864635 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:58.864674 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:58.866370 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:58.867120 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:58.867128 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:58.867757 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:14:58.868320 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:58.868337 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:58.869835 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:58.869839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:58.871223 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:58.871242 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:58.872456 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:58.872462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:58.872673 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:58.873375 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:14:58.873399 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:14:58.875996 (+ 2597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:14:58.878894 (+ 2898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:58.878906 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:58.878923 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:58.879515 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:58.879864 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:58.879871 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:58.879878 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:58.880118 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:58.880621 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:14:58.880631 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:14:58.881075 (+ 444us) client_negotiation.cc:769] Sending connection context
0620 23:14:58.881395 (+ 320us) client_negotiation.cc:240] Negotiation successful
0620 23:14:58.881969 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":296,"threads_started":1}
I20260620 23:14:58.884979 27401 heartbeater.cc:347] Connected to a master server at 127.10.191.254:32873
I20260620 23:14:58.885299 27401 heartbeater.cc:464] Registering TS with master...
I20260620 23:14:58.886169 27401 heartbeater.cc:511] Master 127.10.191.254:32873 requested a full tablet report, sending...
I20260620 23:14:58.888065 27084 ts_manager.cc:194] Registered new tserver with Master: 30bbb7d7a0764e89aaf87e5609420583 (127.10.191.194:37623)
I20260620 23:14:58.889197 27084 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.194@KRBTEST.COM'} at 127.10.191.194:44311
W20260620 23:14:59.192111 27404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:14:59.192385 27404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:14:59.192479 27404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:14:59.199568 27404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:14:59.199669 27404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:14:59.199779 27404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.195
I20260620 23:14:59.209424 27404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:33627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.195:33237
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.191.195
--webserver_port=35375
--tserver_master_addrs=127.10.191.254:32873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.195
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:14:59.210680 27404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:14:59.212015 27404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:14:59.225874 27410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:59.226186 27409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:14:59.226323 27412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:14:59.226799 27404 server_base.cc:1034] running on GCE node
I20260620 23:14:59.227473 27404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:14:59.228392 27404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:14:59.229588 27404 hybrid_clock.cc:648] HybridClock initialized: now 1781997299229530 us; error 52 us; skew 500 ppm
I20260620 23:14:59.229899 27404 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:14:59 dist-test-slave-jkbq krb5kdc[26500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997299, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:14:59.233191 27404 init.cc:377] Logged in from keytab as kudu/127.10.191.195@KRBTEST.COM (short username kudu)
I20260620 23:14:59.234686 27404 webserver.cc:460] Webserver started at http://127.10.191.195:35375/ using document root <none> and password file <none>
I20260620 23:14:59.235412 27404 fs_manager.cc:359] Metadata directory not provided
I20260620 23:14:59.235594 27404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:14:59.241039 27404 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 23:14:59.243677 27419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:59.244904 27404 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:14:59.245079 27404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/wal
uuid: "714cc984998f43f4b0ea18c70fa7382d"
format_stamp: "Formatted at 2026-06-20 23:14:36 on dist-test-slave-jkbq"
server_key: "2305fd1c25c65057b377af8592c2f6d4"
server_key_iv: "5a234915dde2c4e6e0084737c6e6d608"
server_key_version: "encryptionkey@0"
I20260620 23:14:59.246026 27404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:14:59.274257 27404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:14:59.278589 27404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:14:59.278914 27404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:14:59.279922 27404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:14:59.281481 27404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:14:59.281620 27404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:59.281740 27404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:14:59.281827 27404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:14:59.314546 27404 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.195:33237
I20260620 23:14:59.314633 27531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.195:33237 every 8 connection(s)
I20260620 23:14:59.316037 27404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
I20260620 23:14:59.323505 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 27404
Jun 20 23:14:59 dist-test-slave-jkbq krb5kdc[26500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997299, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:14:59.336829 27271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:59.318502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:38023 (local address 127.10.191.254:32873)
0620 23:14:59.318727 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:59.318734 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:59.319940 (+ 1206us) server_negotiation.cc:407] Connection header received
0620 23:14:59.321722 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:59.321727 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:59.321795 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:59.321885 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:59.323443 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:59.324459 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:59.326294 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:59.327063 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:59.327208 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:59.330429 (+ 3221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:59.330435 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:59.330438 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:59.330470 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:59.333267 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:59.334146 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:59.334152 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:59.334155 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:59.334283 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:59.334830 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:59.334835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:59.334838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:59.335222 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:59.335342 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:59.336231 (+ 889us) server_negotiation.cc:299] Negotiation successful
0620 23:14:59.336462 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 23:14:59.337950 27534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:59.319100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.195:38023)
0620 23:14:59.319780 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:59.319821 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:59.321475 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:59.322093 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:59.322100 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:59.322629 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:14:59.323266 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:59.323283 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:59.324640 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:59.324647 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:59.326154 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:59.326165 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:59.327264 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:59.327269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:59.327460 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:59.328120 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:14:59.328142 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:14:59.330212 (+ 2070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:14:59.333436 (+ 3224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:59.333444 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:59.333461 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:59.334021 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:59.334423 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:14:59.334427 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:14:59.334429 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:14:59.334691 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:14:59.335355 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:14:59.335365 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:14:59.335806 (+ 441us) client_negotiation.cc:769] Sending connection context
0620 23:14:59.336139 (+ 333us) client_negotiation.cc:240] Negotiation successful
0620 23:14:59.336605 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":263,"threads_started":1}
I20260620 23:14:59.339483 27535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:59.324476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43888 (local address 127.10.191.254:32873)
0620 23:14:59.324978 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:59.324982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:59.325413 (+ 431us) server_negotiation.cc:407] Connection header received
0620 23:14:59.325600 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:59.325605 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:59.325669 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:59.325775 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:59.327778 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:59.328493 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:59.329920 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:59.330313 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:59.330444 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:59.333055 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:59.333060 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:59.333063 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:59.333093 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:59.336087 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:59.337325 (+ 1238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:59.337337 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:59.337341 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:59.337475 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:59.337837 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:59.337842 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:59.337845 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:59.338207 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:59.338454 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:59.338854 (+ 400us) server_negotiation.cc:299] Negotiation successful
0620 23:14:59.339117 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":280,"threads_started":1}
I20260620 23:14:59.339582 27532 heartbeater.cc:347] Connected to a master server at 127.10.191.254:32873
I20260620 23:14:59.339905 27532 heartbeater.cc:464] Registering TS with master...
I20260620 23:14:59.340787 27532 heartbeater.cc:511] Master 127.10.191.254:32873 requested a full tablet report, sending...
I20260620 23:14:59.342222 27084 ts_manager.cc:194] Registered new tserver with Master: 714cc984998f43f4b0ea18c70fa7382d (127.10.191.195:33237)
I20260620 23:14:59.343087 27084 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.195@KRBTEST.COM'} at 127.10.191.195:38023
I20260620 23:14:59.354161 11007 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:14:59.360347 27535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:59.355063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43898 (local address 127.10.191.254:32873)
0620 23:14:59.355222 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:59.355226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:59.355681 (+ 455us) server_negotiation.cc:407] Connection header received
0620 23:14:59.355867 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:59.355877 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:59.356026 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:59.356148 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:14:59.356550 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:59.357243 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:59.358668 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:59.359140 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:59.359230 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:59.359679 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:14:59.359865 (+ 186us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 23:14:59.360140 (+ 275us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43898: token expired
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 23:14:59.360450 27535 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43898: token expired
W20260620 23:14:59.361016 27007 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.10.191.254:32873)
I20260620 23:14:59.373317 27535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:59.361565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43906 (local address 127.10.191.254:32873)
0620 23:14:59.361714 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:59.361718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:59.361733 (+ 15us) server_negotiation.cc:407] Connection header received
0620 23:14:59.361794 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:59.361799 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:59.361866 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:59.361940 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:59.363446 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:59.364134 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:59.365489 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:59.365870 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:59.365963 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:59.368467 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:59.368471 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:59.368474 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:59.368502 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:59.370820 (+ 2318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:59.371556 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:59.371560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:59.371563 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:59.371697 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:59.371997 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:59.372000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:59.372002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:59.372392 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:59.372633 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:59.372812 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 23:14:59.372977 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 23:14:59.376590 27084 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43906:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:14:59.383944 27084 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:14:59.408496 27543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:59.395869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:33111 (local address 127.0.0.1:42632)
0620 23:14:59.396938 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:59.396975 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:59.397152 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:59.398237 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:59.398246 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:59.398277 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:14:59.398643 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:59.398656 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:59.400693 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:59.400699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:59.404267 (+ 3568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:59.404280 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:59.407488 (+ 3208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:59.407496 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:59.407699 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:59.407748 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 23:14:59.407875 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 23:14:59.408063 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":515,"threads_started":1}
I20260620 23:14:59.408999 27544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:59.397155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.195:33237 (local address 127.0.0.1:34734)
0620 23:14:59.397710 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:59.397726 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:59.397829 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:59.401236 (+ 3407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:59.401241 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:59.401265 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:14:59.401661 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:59.401673 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:59.403927 (+ 2254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:59.403933 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:59.406099 (+ 2166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:59.406117 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:59.408296 (+ 2179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:59.408302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:59.408471 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:59.408513 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 23:14:59.408579 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 23:14:59.408696 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":299,"threads_started":1}
I20260620 23:14:59.409195 27542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:59.395926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42632 (local address 127.10.191.193:33111)
0620 23:14:59.396494 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:59.396502 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:59.397073 (+ 571us) server_negotiation.cc:407] Connection header received
0620 23:14:59.397440 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:59.397449 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:59.397710 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:59.397953 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:14:59.398857 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:59.400564 (+ 1707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:59.404470 (+ 3906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:59.405950 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:59.406152 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:59.407320 (+ 1168us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:59.408710 (+ 1390us) server_negotiation.cc:299] Negotiation successful
0620 23:14:59.408881 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":259,"threads_started":1}
I20260620 23:14:59.409936 27545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:59.397503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34734 (local address 127.10.191.195:33237)
0620 23:14:59.398403 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:59.398414 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:59.398449 (+ 35us) server_negotiation.cc:407] Connection header received
0620 23:14:59.400765 (+ 2316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:59.400773 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:59.401026 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:59.401856 (+ 830us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:14:59.402174 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:59.403765 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:59.406627 (+ 2862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:59.408030 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:59.409039 (+ 1009us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:59.409212 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:59.409285 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 23:14:59.409588 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":744,"thread_start_us":573,"threads_started":1}
I20260620 23:14:59.413152 27204 tablet_service.cc:1467] Processing CreateTablet for tablet 05b2d6545c454f9a8a1a41fe8b78cb09 (DEFAULT_TABLE table=test-table [id=e28420aba43d48c59e9410707a57aef3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:14:59.413601 27467 tablet_service.cc:1467] Processing CreateTablet for tablet 05b2d6545c454f9a8a1a41fe8b78cb09 (DEFAULT_TABLE table=test-table [id=e28420aba43d48c59e9410707a57aef3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:14:59.414453 27204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05b2d6545c454f9a8a1a41fe8b78cb09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:14:59.414956 27467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05b2d6545c454f9a8a1a41fe8b78cb09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:14:59.416822 27546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:59.397784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.194:37623 (local address 127.0.0.1:59732)
0620 23:14:59.408406 (+ 10622us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:59.408423 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:59.409225 (+ 802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:59.409930 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:59.409936 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:59.409964 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:14:59.410447 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:59.410459 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:59.412352 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:59.412357 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:59.414276 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:59.414292 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:59.416078 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:59.416084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:59.416246 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:59.416279 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 23:14:59.416389 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 23:14:59.416516 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10486,"thread_start_us":11027,"threads_started":1}
I20260620 23:14:59.416947 27547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:59.398274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59732 (local address 127.10.191.194:37623)
0620 23:14:59.398835 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:59.398843 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:59.408529 (+ 9686us) server_negotiation.cc:407] Connection header received
0620 23:14:59.409475 (+ 946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:59.409485 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:59.409722 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:59.409927 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:14:59.410662 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:59.412211 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:59.414445 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:59.415845 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:59.416018 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:59.416165 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:59.416460 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 23:14:59.416612 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":278,"threads_started":1}
I20260620 23:14:59.422420 27336 tablet_service.cc:1467] Processing CreateTablet for tablet 05b2d6545c454f9a8a1a41fe8b78cb09 (DEFAULT_TABLE table=test-table [id=e28420aba43d48c59e9410707a57aef3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:14:59.423753 27336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05b2d6545c454f9a8a1a41fe8b78cb09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:14:59.427098 27548 tablet_bootstrap.cc:492] T 05b2d6545c454f9a8a1a41fe8b78cb09 P c5ff4a994ed24d10ae1b3b1f8fef3789: Bootstrap starting.
I20260620 23:14:59.429517 27549 tablet_bootstrap.cc:492] T 05b2d6545c454f9a8a1a41fe8b78cb09 P 714cc984998f43f4b0ea18c70fa7382d: Bootstrap starting.
I20260620 23:14:59.430600 27548 tablet_bootstrap.cc:654] T 05b2d6545c454f9a8a1a41fe8b78cb09 P c5ff4a994ed24d10ae1b3b1f8fef3789: Neither blocks nor log segments found. Creating new log.
I20260620 23:14:59.431900 27548 log.cc:824] T 05b2d6545c454f9a8a1a41fe8b78cb09 P c5ff4a994ed24d10ae1b3b1f8fef3789: Log is configured to *not* fsync() on all Append() calls
I20260620 23:14:59.432932 27549 tablet_bootstrap.cc:654] T 05b2d6545c454f9a8a1a41fe8b78cb09 P 714cc984998f43f4b0ea18c70fa7382d: Neither blocks nor log segments found. Creating new log.
I20260620 23:14:59.434170 27549 log.cc:824] T 05b2d6545c454f9a8a1a41fe8b78cb09 P 714cc984998f43f4b0ea18c70fa7382d: Log is configured to *not* fsync() on all Append() calls
I20260620 23:14:59.435086 27548 tablet_bootstrap.cc:492] T 05b2d6545c454f9a8a1a41fe8b78cb09 P c5ff4a994ed24d10ae1b3b1f8fef3789: No bootstrap required, opened a new log
I20260620 23:14:59.435361 27548 ts_tablet_manager.cc:1397] T 05b2d6545c454f9a8a1a41fe8b78cb09 P c5ff4a994ed24d10ae1b3b1f8fef3789: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 23:14:59.437031 27549 tablet_bootstrap.cc:492] T 05b2d6545c454f9a8a1a41fe8b78cb09 P 714cc984998f43f4b0ea18c70fa7382d: No bootstrap required, opened a new log
I20260620 23:14:59.437304 27549 ts_tablet_manager.cc:1397] T 05b2d6545c454f9a8a1a41fe8b78cb09 P 714cc984998f43f4b0ea18c70fa7382d: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260620 23:14:59.438822 27269 heartbeater.cc:503] Master 127.10.191.254:32873 was elected leader, sending a full tablet report...
I20260620 23:14:59.439060 27552 tablet_bootstrap.cc:492] T 05b2d6545c454f9a8a1a41fe8b78cb09 P 30bbb7d7a0764e89aaf87e5609420583: Bootstrap starting.
I20260620 23:14:59.442479 27552 tablet_bootstrap.cc:654] T 05b2d6545c454f9a8a1a41fe8b78cb09 P 30bbb7d7a0764e89aaf87e5609420583: Neither blocks nor log segments found. Creating new log.
I20260620 23:14:59.443936 27552 log.cc:824] T 05b2d6545c454f9a8a1a41fe8b78cb09 P 30bbb7d7a0764e89aaf87e5609420583: Log is configured to *not* fsync() on all Append() calls
I20260620 23:14:59.446273 27548 raft_consensus.cc:348] T 05b2d6545c454f9a8a1a41fe8b78cb09 P c5ff4a994ed24d10ae1b3b1f8fef3789 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ff4a994ed24d10ae1b3b1f8fef3789" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 33111 } } peers { permanent_uuid: "714cc984998f43f4b0ea18c70fa7382d" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 33237 } } peers { permanent_uuid: "30bbb7d7a0764e89aaf87e5609420583" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37623 } }
I20260620 23:14:59.446640 27548 raft_consensus.cc:374] T 05b2d6545c454f9a8a1a41fe8b78cb09 P c5ff4a994ed24d10ae1b3b1f8fef3789 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:14:59.446754 27548 raft_consensus.cc:729] T 05b2d6545c454f9a8a1a41fe8b78cb09 P c5ff4a994ed24d10ae1b3b1f8fef3789 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ff4a994ed24d10ae1b3b1f8fef3789, State: Initialized, Role: FOLLOWER
I20260620 23:14:59.447578 27548 consensus_queue.cc:260] T 05b2d6545c454f9a8a1a41fe8b78cb09 P c5ff4a994ed24d10ae1b3b1f8fef3789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ff4a994ed24d10ae1b3b1f8fef3789" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 33111 } } peers { permanent_uuid: "714cc984998f43f4b0ea18c70fa7382d" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 33237 } } peers { permanent_uuid: "30bbb7d7a0764e89aaf87e5609420583" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37623 } }
I20260620 23:14:59.448621 27549 raft_consensus.cc:348] T 05b2d6545c454f9a8a1a41fe8b78cb09 P 714cc984998f43f4b0ea18c70fa7382d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ff4a994ed24d10ae1b3b1f8fef3789" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 33111 } } peers { permanent_uuid: "714cc984998f43f4b0ea18c70fa7382d" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 33237 } } peers { permanent_uuid: "30bbb7d7a0764e89aaf87e5609420583" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37623 } }
I20260620 23:14:59.449079 27549 raft_consensus.cc:374] T 05b2d6545c454f9a8a1a41fe8b78cb09 P 714cc984998f43f4b0ea18c70fa7382d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:14:59.449177 27549 raft_consensus.cc:729] T 05b2d6545c454f9a8a1a41fe8b78cb09 P 714cc984998f43f4b0ea18c70fa7382d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 714cc984998f43f4b0ea18c70fa7382d, State: Initialized, Role: FOLLOWER
I20260620 23:14:59.449930 27548 ts_tablet_manager.cc:1428] T 05b2d6545c454f9a8a1a41fe8b78cb09 P c5ff4a994ed24d10ae1b3b1f8fef3789: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260620 23:14:59.449833 27549 consensus_queue.cc:260] T 05b2d6545c454f9a8a1a41fe8b78cb09 P 714cc984998f43f4b0ea18c70fa7382d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ff4a994ed24d10ae1b3b1f8fef3789" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 33111 } } peers { permanent_uuid: "714cc984998f43f4b0ea18c70fa7382d" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 33237 } } peers { permanent_uuid: "30bbb7d7a0764e89aaf87e5609420583" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37623 } }
I20260620 23:14:59.450516 27552 tablet_bootstrap.cc:492] T 05b2d6545c454f9a8a1a41fe8b78cb09 P 30bbb7d7a0764e89aaf87e5609420583: No bootstrap required, opened a new log
I20260620 23:14:59.450801 27552 ts_tablet_manager.cc:1397] T 05b2d6545c454f9a8a1a41fe8b78cb09 P 30bbb7d7a0764e89aaf87e5609420583: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260620 23:14:59.451637 27532 heartbeater.cc:503] Master 127.10.191.254:32873 was elected leader, sending a full tablet report...
I20260620 23:14:59.451733 27549 ts_tablet_manager.cc:1428] T 05b2d6545c454f9a8a1a41fe8b78cb09 P 714cc984998f43f4b0ea18c70fa7382d: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260620 23:14:59.460016 27552 raft_consensus.cc:348] T 05b2d6545c454f9a8a1a41fe8b78cb09 P 30bbb7d7a0764e89aaf87e5609420583 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ff4a994ed24d10ae1b3b1f8fef3789" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 33111 } } peers { permanent_uuid: "714cc984998f43f4b0ea18c70fa7382d" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 33237 } } peers { permanent_uuid: "30bbb7d7a0764e89aaf87e5609420583" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37623 } }
I20260620 23:14:59.460369 27552 raft_consensus.cc:374] T 05b2d6545c454f9a8a1a41fe8b78cb09 P 30bbb7d7a0764e89aaf87e5609420583 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:14:59.460507 27552 raft_consensus.cc:729] T 05b2d6545c454f9a8a1a41fe8b78cb09 P 30bbb7d7a0764e89aaf87e5609420583 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30bbb7d7a0764e89aaf87e5609420583, State: Initialized, Role: FOLLOWER
I20260620 23:14:59.461103 27552 consensus_queue.cc:260] T 05b2d6545c454f9a8a1a41fe8b78cb09 P 30bbb7d7a0764e89aaf87e5609420583 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ff4a994ed24d10ae1b3b1f8fef3789" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 33111 } } peers { permanent_uuid: "714cc984998f43f4b0ea18c70fa7382d" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 33237 } } peers { permanent_uuid: "30bbb7d7a0764e89aaf87e5609420583" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37623 } }
I20260620 23:14:59.462121 27401 heartbeater.cc:503] Master 127.10.191.254:32873 was elected leader, sending a full tablet report...
I20260620 23:14:59.462813 27552 ts_tablet_manager.cc:1428] T 05b2d6545c454f9a8a1a41fe8b78cb09 P 30bbb7d7a0764e89aaf87e5609420583: Time spent starting tablet: real 0.012s user 0.009s sys 0.001s
W20260620 23:14:59.567682 27533 tablet.cc:2369] T 05b2d6545c454f9a8a1a41fe8b78cb09 P 714cc984998f43f4b0ea18c70fa7382d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:14:59.612556 27402 tablet.cc:2369] T 05b2d6545c454f9a8a1a41fe8b78cb09 P 30bbb7d7a0764e89aaf87e5609420583: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:14:59.653283 27270 tablet.cc:2369] T 05b2d6545c454f9a8a1a41fe8b78cb09 P c5ff4a994ed24d10ae1b3b1f8fef3789: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:14:59.727383 27554 raft_consensus.cc:482] T 05b2d6545c454f9a8a1a41fe8b78cb09 P c5ff4a994ed24d10ae1b3b1f8fef3789 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:14:59.727751 27554 raft_consensus.cc:504] T 05b2d6545c454f9a8a1a41fe8b78cb09 P c5ff4a994ed24d10ae1b3b1f8fef3789 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ff4a994ed24d10ae1b3b1f8fef3789" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 33111 } } peers { permanent_uuid: "714cc984998f43f4b0ea18c70fa7382d" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 33237 } } peers { permanent_uuid: "30bbb7d7a0764e89aaf87e5609420583" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37623 } }
I20260620 23:14:59.729511 27554 leader_election.cc:288] T 05b2d6545c454f9a8a1a41fe8b78cb09 P c5ff4a994ed24d10ae1b3b1f8fef3789 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 714cc984998f43f4b0ea18c70fa7382d (127.10.191.195:33237), 30bbb7d7a0764e89aaf87e5609420583 (127.10.191.194:37623)
I20260620 23:14:59.736289 27545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:59.729960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:48163 (local address 127.10.191.195:33237)
0620 23:14:59.730155 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:59.730159 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:59.730956 (+ 797us) server_negotiation.cc:407] Connection header received
0620 23:14:59.731046 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:59.731051 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:59.731136 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:59.731256 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:14:59.731931 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:59.732642 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:59.734420 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:59.735352 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:59.735543 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:59.735617 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:59.735883 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 23:14:59.736013 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:14:59.736452 27557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:59.729972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.195:33237 (local address 127.10.191.193:48163)
0620 23:14:59.730700 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:59.730721 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:59.730894 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:59.731431 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:59.731441 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:59.731466 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:14:59.731776 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:59.731784 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:59.732786 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:59.732794 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:59.734256 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:59.734269 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:59.735558 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:59.735564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:59.735722 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:59.735748 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 23:14:59.735848 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 23:14:59.736030 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"mutex_wait_us":3,"thread_start_us":421,"threads_started":1}
I20260620 23:14:59.737180 27558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:59.730854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.194:37623 (local address 127.10.191.193:37113)
0620 23:14:59.731599 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:59.731616 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:14:59.731775 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:14:59.732097 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:14:59.732101 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:14:59.732124 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:14:59.732452 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:59.732460 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:59.733476 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:59.733480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:59.734799 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:14:59.734810 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:14:59.736067 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:14:59.736073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:14:59.736266 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:59.736290 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 23:14:59.736733 (+ 443us) client_negotiation.cc:240] Negotiation successful
0620 23:14:59.736862 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":385,"threads_started":1}
I20260620 23:14:59.737187 27547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:59.731073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:37113 (local address 127.10.191.194:37623)
0620 23:14:59.731335 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:59.731341 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:59.731704 (+ 363us) server_negotiation.cc:407] Connection header received
0620 23:14:59.731922 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:59.731927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:59.731985 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:59.732081 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:14:59.732598 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:59.733313 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:59.734986 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:59.735831 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:59.736041 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:59.736113 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:59.736806 (+ 693us) server_negotiation.cc:299] Negotiation successful
0620 23:14:59.736943 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 23:14:59.737480 27487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05b2d6545c454f9a8a1a41fe8b78cb09" candidate_uuid: "c5ff4a994ed24d10ae1b3b1f8fef3789" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "714cc984998f43f4b0ea18c70fa7382d" is_pre_election: true
I20260620 23:14:59.738121 27487 raft_consensus.cc:2454] T 05b2d6545c454f9a8a1a41fe8b78cb09 P 714cc984998f43f4b0ea18c70fa7382d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ff4a994ed24d10ae1b3b1f8fef3789 in term 0.
I20260620 23:14:59.738132 27356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05b2d6545c454f9a8a1a41fe8b78cb09" candidate_uuid: "c5ff4a994ed24d10ae1b3b1f8fef3789" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30bbb7d7a0764e89aaf87e5609420583" is_pre_election: true
I20260620 23:14:59.738590 27356 raft_consensus.cc:2454] T 05b2d6545c454f9a8a1a41fe8b78cb09 P 30bbb7d7a0764e89aaf87e5609420583 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ff4a994ed24d10ae1b3b1f8fef3789 in term 0.
I20260620 23:14:59.738837 27158 leader_election.cc:302] T 05b2d6545c454f9a8a1a41fe8b78cb09 P c5ff4a994ed24d10ae1b3b1f8fef3789 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 714cc984998f43f4b0ea18c70fa7382d, c5ff4a994ed24d10ae1b3b1f8fef3789; no voters:
I20260620 23:14:59.739271 27554 raft_consensus.cc:2781] T 05b2d6545c454f9a8a1a41fe8b78cb09 P c5ff4a994ed24d10ae1b3b1f8fef3789 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:14:59.739421 27554 raft_consensus.cc:482] T 05b2d6545c454f9a8a1a41fe8b78cb09 P c5ff4a994ed24d10ae1b3b1f8fef3789 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:14:59.739549 27554 raft_consensus.cc:3037] T 05b2d6545c454f9a8a1a41fe8b78cb09 P c5ff4a994ed24d10ae1b3b1f8fef3789 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:14:59.741753 27554 raft_consensus.cc:504] T 05b2d6545c454f9a8a1a41fe8b78cb09 P c5ff4a994ed24d10ae1b3b1f8fef3789 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ff4a994ed24d10ae1b3b1f8fef3789" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 33111 } } peers { permanent_uuid: "714cc984998f43f4b0ea18c70fa7382d" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 33237 } } peers { permanent_uuid: "30bbb7d7a0764e89aaf87e5609420583" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37623 } }
I20260620 23:14:59.742333 27554 leader_election.cc:288] T 05b2d6545c454f9a8a1a41fe8b78cb09 P c5ff4a994ed24d10ae1b3b1f8fef3789 [CANDIDATE]: Term 1 election: Requested vote from peers 714cc984998f43f4b0ea18c70fa7382d (127.10.191.195:33237), 30bbb7d7a0764e89aaf87e5609420583 (127.10.191.194:37623)
I20260620 23:14:59.742928 27487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05b2d6545c454f9a8a1a41fe8b78cb09" candidate_uuid: "c5ff4a994ed24d10ae1b3b1f8fef3789" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "714cc984998f43f4b0ea18c70fa7382d"
I20260620 23:14:59.743065 27356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05b2d6545c454f9a8a1a41fe8b78cb09" candidate_uuid: "c5ff4a994ed24d10ae1b3b1f8fef3789" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30bbb7d7a0764e89aaf87e5609420583"
I20260620 23:14:59.743177 27487 raft_consensus.cc:3037] T 05b2d6545c454f9a8a1a41fe8b78cb09 P 714cc984998f43f4b0ea18c70fa7382d [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:14:59.743286 27356 raft_consensus.cc:3037] T 05b2d6545c454f9a8a1a41fe8b78cb09 P 30bbb7d7a0764e89aaf87e5609420583 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:14:59.745419 27487 raft_consensus.cc:2454] T 05b2d6545c454f9a8a1a41fe8b78cb09 P 714cc984998f43f4b0ea18c70fa7382d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ff4a994ed24d10ae1b3b1f8fef3789 in term 1.
I20260620 23:14:59.745465 27356 raft_consensus.cc:2454] T 05b2d6545c454f9a8a1a41fe8b78cb09 P 30bbb7d7a0764e89aaf87e5609420583 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ff4a994ed24d10ae1b3b1f8fef3789 in term 1.
I20260620 23:14:59.745810 27158 leader_election.cc:302] T 05b2d6545c454f9a8a1a41fe8b78cb09 P c5ff4a994ed24d10ae1b3b1f8fef3789 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 714cc984998f43f4b0ea18c70fa7382d, c5ff4a994ed24d10ae1b3b1f8fef3789; no voters:
I20260620 23:14:59.746100 27554 raft_consensus.cc:2781] T 05b2d6545c454f9a8a1a41fe8b78cb09 P c5ff4a994ed24d10ae1b3b1f8fef3789 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:14:59.746551 27554 raft_consensus.cc:686] T 05b2d6545c454f9a8a1a41fe8b78cb09 P c5ff4a994ed24d10ae1b3b1f8fef3789 [term 1 LEADER]: Becoming Leader. State: Replica: c5ff4a994ed24d10ae1b3b1f8fef3789, State: Running, Role: LEADER
I20260620 23:14:59.747143 27554 consensus_queue.cc:237] T 05b2d6545c454f9a8a1a41fe8b78cb09 P c5ff4a994ed24d10ae1b3b1f8fef3789 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ff4a994ed24d10ae1b3b1f8fef3789" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 33111 } } peers { permanent_uuid: "714cc984998f43f4b0ea18c70fa7382d" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 33237 } } peers { permanent_uuid: "30bbb7d7a0764e89aaf87e5609420583" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37623 } }
I20260620 23:14:59.751801 27084 catalog_manager.cc:5495] T 05b2d6545c454f9a8a1a41fe8b78cb09 P c5ff4a994ed24d10ae1b3b1f8fef3789 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5ff4a994ed24d10ae1b3b1f8fef3789 (127.10.191.193). New cstate: current_term: 1 leader_uuid: "c5ff4a994ed24d10ae1b3b1f8fef3789" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ff4a994ed24d10ae1b3b1f8fef3789" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 33111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "714cc984998f43f4b0ea18c70fa7382d" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30bbb7d7a0764e89aaf87e5609420583" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37623 } health_report { overall_health: UNKNOWN } } }
I20260620 23:14:59.770028 27542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:59.764575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42634 (local address 127.10.191.193:33111)
0620 23:14:59.764790 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:59.764796 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:59.764813 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:14:59.765025 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:59.765030 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:59.765119 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:59.765274 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:14:59.765643 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:59.766368 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:59.767898 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:59.768311 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:59.768463 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:59.768946 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:14:59.769457 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:14:59.769621 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:59.769677 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 23:14:59.769756 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 23:14:59.778179 27487 raft_consensus.cc:1261] T 05b2d6545c454f9a8a1a41fe8b78cb09 P 714cc984998f43f4b0ea18c70fa7382d [term 1 FOLLOWER]: Refusing update from remote peer c5ff4a994ed24d10ae1b3b1f8fef3789: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:14:59.778182 27356 raft_consensus.cc:1261] T 05b2d6545c454f9a8a1a41fe8b78cb09 P 30bbb7d7a0764e89aaf87e5609420583 [term 1 FOLLOWER]: Refusing update from remote peer c5ff4a994ed24d10ae1b3b1f8fef3789: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:14:59.779181 27559 consensus_queue.cc:1035] T 05b2d6545c454f9a8a1a41fe8b78cb09 P c5ff4a994ed24d10ae1b3b1f8fef3789 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30bbb7d7a0764e89aaf87e5609420583" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 37623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:14:59.779559 27554 consensus_queue.cc:1035] T 05b2d6545c454f9a8a1a41fe8b78cb09 P c5ff4a994ed24d10ae1b3b1f8fef3789 [LEADER]: Connected to new peer: Peer: permanent_uuid: "714cc984998f43f4b0ea18c70fa7382d" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:14:59.805323 27083 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:14:59.805663 27083 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:14:59.809719 27083 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3: Sending DeleteTablet for 3 replicas of tablet 05b2d6545c454f9a8a1a41fe8b78cb09
I20260620 23:14:59.810845 27204 tablet_service.cc:1514] Processing DeleteTablet for tablet 05b2d6545c454f9a8a1a41fe8b78cb09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:14:59 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:42632
I20260620 23:14:59.811156 27467 tablet_service.cc:1514] Processing DeleteTablet for tablet 05b2d6545c454f9a8a1a41fe8b78cb09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:14:59 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:34734
I20260620 23:14:59.811416 27336 tablet_service.cc:1514] Processing DeleteTablet for tablet 05b2d6545c454f9a8a1a41fe8b78cb09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:14:59 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:59732
I20260620 23:14:59.812423 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 27121
I20260620 23:14:59.843019 27574 tablet_replica.cc:330] T 05b2d6545c454f9a8a1a41fe8b78cb09 P 30bbb7d7a0764e89aaf87e5609420583: stopping tablet replica
I20260620 23:14:59.843649 27574 raft_consensus.cc:2229] T 05b2d6545c454f9a8a1a41fe8b78cb09 P 30bbb7d7a0764e89aaf87e5609420583 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:14:59.844290 27574 raft_consensus.cc:2258] T 05b2d6545c454f9a8a1a41fe8b78cb09 P 30bbb7d7a0764e89aaf87e5609420583 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:14:59.845211 27572 tablet_replica.cc:330] T 05b2d6545c454f9a8a1a41fe8b78cb09 P c5ff4a994ed24d10ae1b3b1f8fef3789: stopping tablet replica
I20260620 23:14:59.845623 27572 raft_consensus.cc:2229] T 05b2d6545c454f9a8a1a41fe8b78cb09 P c5ff4a994ed24d10ae1b3b1f8fef3789 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:14:59.846310 27572 raft_consensus.cc:2258] T 05b2d6545c454f9a8a1a41fe8b78cb09 P c5ff4a994ed24d10ae1b3b1f8fef3789 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:14:59.847066 27573 tablet_replica.cc:330] T 05b2d6545c454f9a8a1a41fe8b78cb09 P 714cc984998f43f4b0ea18c70fa7382d: stopping tablet replica
I20260620 23:14:59.847453 27573 raft_consensus.cc:2229] T 05b2d6545c454f9a8a1a41fe8b78cb09 P 714cc984998f43f4b0ea18c70fa7382d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:14:59.847646 27574 ts_tablet_manager.cc:1905] T 05b2d6545c454f9a8a1a41fe8b78cb09 P 30bbb7d7a0764e89aaf87e5609420583: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:14:59.847926 27573 raft_consensus.cc:2258] T 05b2d6545c454f9a8a1a41fe8b78cb09 P 714cc984998f43f4b0ea18c70fa7382d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:14:59.848489 27572 ts_tablet_manager.cc:1905] T 05b2d6545c454f9a8a1a41fe8b78cb09 P c5ff4a994ed24d10ae1b3b1f8fef3789: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:14:59.849906 27573 ts_tablet_manager.cc:1905] T 05b2d6545c454f9a8a1a41fe8b78cb09 P 714cc984998f43f4b0ea18c70fa7382d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:14:59.853493 27542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:59.813291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42640 (local address 127.10.191.193:33111)
0620 23:14:59.813897 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:59.813906 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:59.813932 (+ 26us) server_negotiation.cc:407] Connection header received
0620 23:14:59.814272 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:59.814280 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:59.814421 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:59.814547 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:59.817192 (+ 2645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:59.818794 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:59.822187 (+ 3393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:59.823180 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:59.823334 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:59.833555 (+ 10221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:59.833577 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:59.833605 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:59.833695 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:59.839808 (+ 6113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:59.850954 (+ 11146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:59.850963 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:59.850966 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:59.851240 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:59.851729 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:59.851734 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:59.851738 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:59.852153 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:59.852462 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:59.852821 (+ 359us) server_negotiation.cc:299] Negotiation successful
0620 23:14:59.853085 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451}
I20260620 23:14:59.854250 27264 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:14:59.858768 27574 ts_tablet_manager.cc:1918] T 05b2d6545c454f9a8a1a41fe8b78cb09 P 30bbb7d7a0764e89aaf87e5609420583: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:14:59.858958 27574 log.cc:1196] T 05b2d6545c454f9a8a1a41fe8b78cb09 P 30bbb7d7a0764e89aaf87e5609420583: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/wal/wals/05b2d6545c454f9a8a1a41fe8b78cb09
I20260620 23:14:59.859262 27573 ts_tablet_manager.cc:1918] T 05b2d6545c454f9a8a1a41fe8b78cb09 P 714cc984998f43f4b0ea18c70fa7382d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:14:59.859416 27573 log.cc:1196] T 05b2d6545c454f9a8a1a41fe8b78cb09 P 714cc984998f43f4b0ea18c70fa7382d: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/wal/wals/05b2d6545c454f9a8a1a41fe8b78cb09
I20260620 23:14:59.859484 27574 ts_tablet_manager.cc:1939] T 05b2d6545c454f9a8a1a41fe8b78cb09 P 30bbb7d7a0764e89aaf87e5609420583: Deleting consensus metadata
I20260620 23:14:59.859870 27573 ts_tablet_manager.cc:1939] T 05b2d6545c454f9a8a1a41fe8b78cb09 P 714cc984998f43f4b0ea18c70fa7382d: Deleting consensus metadata
I20260620 23:14:59.861413 27069 catalog_manager.cc:4841] TS 30bbb7d7a0764e89aaf87e5609420583 (127.10.191.194:37623): tablet 05b2d6545c454f9a8a1a41fe8b78cb09 (table test-table [id=e28420aba43d48c59e9410707a57aef3]) successfully deleted
I20260620 23:14:59.861805 27069 catalog_manager.cc:4841] TS 714cc984998f43f4b0ea18c70fa7382d (127.10.191.195:33237): tablet 05b2d6545c454f9a8a1a41fe8b78cb09 (table test-table [id=e28420aba43d48c59e9410707a57aef3]) successfully deleted
I20260620 23:14:59.953075 27572 ts_tablet_manager.cc:1918] T 05b2d6545c454f9a8a1a41fe8b78cb09 P c5ff4a994ed24d10ae1b3b1f8fef3789: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:14:59.953368 27572 log.cc:1196] T 05b2d6545c454f9a8a1a41fe8b78cb09 P c5ff4a994ed24d10ae1b3b1f8fef3789: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/wal/wals/05b2d6545c454f9a8a1a41fe8b78cb09
I20260620 23:14:59.953863 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 27121
W20260620 23:14:59.971938 27069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.191.193:33111 (error 108)
I20260620 23:14:59.972617 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 27273
I20260620 23:14:59.973886 27577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:59.973165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:33111 (local address 127.0.0.1:42652)
0620 23:14:59.973605 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0620 23:14:59.973715 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.193:33111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":236,"threads_started":1}
W20260620 23:14:59.974176 27069 catalog_manager.cc:4584] TS c5ff4a994ed24d10ae1b3b1f8fef3789 (127.10.191.193:33111): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 05b2d6545c454f9a8a1a41fe8b78cb09: Network error: Client connection negotiation failed: client connection to 127.10.191.193:33111: connect: Connection refused (error 111)
I20260620 23:14:59.986729 27547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:14:59.973408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59736 (local address 127.10.191.194:37623)
0620 23:14:59.973593 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 23:14:59.973599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:14:59.973705 (+ 106us) server_negotiation.cc:407] Connection header received
0620 23:14:59.973923 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:14:59.973930 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:14:59.974027 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:14:59.974157 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:14:59.975624 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:59.976334 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:59.977693 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:14:59.978092 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:14:59.978235 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:14:59.981135 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:14:59.981149 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:14:59.981162 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:14:59.981199 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:14:59.983941 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:59.984637 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:59.984652 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:59.984656 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:59.984783 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:14:59.985163 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:14:59.985169 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:14:59.985172 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:14:59.985545 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:14:59.985836 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0620 23:14:59.986161 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 23:14:59.986393 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 23:14:59.987481 27396 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:15:00.027752 27577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:00.027297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:33111 (local address 127.0.0.1:42668)
0620 23:15:00.027513 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:00.027610 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.193:33111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 23:15:00.028873 27577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:00.028474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:33111 (local address 127.0.0.1:42684)
0620 23:15:00.028654 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:00.028727 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.193:33111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 23:15:00.078078 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 27273
I20260620 23:15:00.089488 27577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:00.089017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:33111 (local address 127.0.0.1:42694)
0620 23:15:00.089243 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:00.089332 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.193:33111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 23:15:00.090672 27577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:00.090264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:33111 (local address 127.0.0.1:42700)
0620 23:15:00.090449 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:00.090523 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.193:33111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 23:15:00.095913 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 27404
I20260620 23:15:00.109005 27545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:00.096660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34742 (local address 127.10.191.195:33237)
0620 23:15:00.096822 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:00.096826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:00.096845 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:15:00.096992 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:00.096996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:00.097061 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:00.097145 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:00.098503 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:00.099288 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:00.100629 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:00.101020 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:00.101110 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:00.103765 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:00.103779 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:00.103796 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:00.103832 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:00.106424 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:00.107177 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:00.107185 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:00.107187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:00.107326 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:00.107650 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:00.107653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:00.107655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:00.108014 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:00.108240 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:00.108529 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 23:15:00.108697 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 23:15:00.109624 27527 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:15:00.171046 27577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:00.170484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:33111 (local address 127.0.0.1:42716)
0620 23:15:00.170719 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:00.170821 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.193:33111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 23:15:00.172097 27577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:00.171677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:33111 (local address 127.0.0.1:42722)
0620 23:15:00.171881 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:00.171959 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.193:33111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260620 23:15:00.199649 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 27404
I20260620 23:15:00.216355 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 27051
I20260620 23:15:00.217494 27114 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:15:00.334403 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 27051
I20260620 23:15:20.356616 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:32873
--webserver_interface=127.10.191.254
--webserver_port=34225
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:33627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:32873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:15:20.705113 27581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:20.705396 27581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:20.705554 27581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:20.712847 27581 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 23:15:20.712939 27581 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 23:15:20.713004 27581 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 23:15:20.713081 27581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:15:20.713173 27581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:20.713284 27581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:20.713356 27581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:15:20.713447 27581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:15:20.723189 27581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:33627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:32873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:32873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=34225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:15:20.724450 27581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:20.725767 27581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:20.737807 27586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:20.737900 27587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:20.737843 27589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:20.739104 27581 server_base.cc:1034] running on GCE node
I20260620 23:15:20.739842 27581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:20.741164 27581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:20.742414 27581 hybrid_clock.cc:648] HybridClock initialized: now 1781997320742344 us; error 55 us; skew 500 ppm
I20260620 23:15:20.742756 27581 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:15:20 dist-test-slave-jkbq krb5kdc[26500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997320, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:20.745941 27581 init.cc:377] Logged in from keytab as kudu/127.10.191.254@KRBTEST.COM (short username kudu)
I20260620 23:15:20.747509 27581 webserver.cc:460] Webserver started at http://127.10.191.254:34225/ using document root <none> and password file <none>
I20260620 23:15:20.748193 27581 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:20.748337 27581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:20.753683 27581 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 23:15:20.756491 27596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:20.757779 27581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:15:20.757997 27581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "d6d9ed6d31094bb29fa59de15a5924a3"
format_stamp: "Formatted at 2026-06-20 23:14:34 on dist-test-slave-jkbq"
server_key: "2cf0710873a8459633bae0f7d6613168"
server_key_iv: "655b8f7dd28bd835c94f275f04379e4f"
server_key_version: "encryptionkey@0"
I20260620 23:15:20.758971 27581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:20.778784 27581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:20.783073 27581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:20.783304 27581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:20.801957 27581 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:32873
I20260620 23:15:20.802060 27647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:32873 every 8 connection(s)
I20260620 23:15:20.803573 27581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:15:20.810407 27648 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:15:20.810586 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 27581
I20260620 23:15:20.811519 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.193:33111
--local_ip_for_outbound_sockets=127.10.191.193
--tserver_master_addrs=127.10.191.254:32873
--webserver_port=45075
--webserver_interface=127.10.191.193
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:33627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:15:20.813798 27648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3: Bootstrap starting.
I20260620 23:15:20.833879 27648 log.cc:824] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:20.849937 27648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:15:20.850498 27648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3: Bootstrap complete.
I20260620 23:15:20.859505 27648 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d9ed6d31094bb29fa59de15a5924a3" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 32873 } }
I20260620 23:15:20.860428 27648 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6d9ed6d31094bb29fa59de15a5924a3, State: Initialized, Role: FOLLOWER
I20260620 23:15:20.861120 27648 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d9ed6d31094bb29fa59de15a5924a3" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 32873 } }
I20260620 23:15:20.861383 27648 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:15:20.861511 27648 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:15:20.861717 27648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [term 2 FOLLOWER]: Advancing to term 3
I20260620 23:15:20.865911 27648 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d9ed6d31094bb29fa59de15a5924a3" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 32873 } }
I20260620 23:15:20.866447 27648 leader_election.cc:302] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6d9ed6d31094bb29fa59de15a5924a3; no voters:
I20260620 23:15:20.867208 27648 leader_election.cc:288] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 23:15:20.867416 27652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 23:15:20.868583 27652 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [term 3 LEADER]: Becoming Leader. State: Replica: d6d9ed6d31094bb29fa59de15a5924a3, State: Running, Role: LEADER
I20260620 23:15:20.869153 27652 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d9ed6d31094bb29fa59de15a5924a3" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 32873 } }
I20260620 23:15:20.869498 27648 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:15:20.871730 27654 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6d9ed6d31094bb29fa59de15a5924a3. Latest consensus state: current_term: 3 leader_uuid: "d6d9ed6d31094bb29fa59de15a5924a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d9ed6d31094bb29fa59de15a5924a3" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 32873 } } }
I20260620 23:15:20.872191 27654 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [sys.catalog]: This master's current role is: LEADER
I20260620 23:15:20.872839 27653 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d6d9ed6d31094bb29fa59de15a5924a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d9ed6d31094bb29fa59de15a5924a3" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 32873 } } }
I20260620 23:15:20.877401 27653 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3 [sys.catalog]: This master's current role is: LEADER
I20260620 23:15:20.878454 27660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:15:20.889386 27660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:15:20.891530 27660 catalog_manager.cc:1259] Loaded cluster ID: a365fdc97ce44b7aaa187d49cea26635
I20260620 23:15:20.891732 27660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:15:20.896070 27669 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 23:15:20.896883 27660 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:15:20.898190 27660 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3: Loaded TSK: 0
I20260620 23:15:20.898330 27660 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d6d9ed6d31094bb29fa59de15a5924a3: Loaded TSK: 1
F20260620 23:15:20.899205 27660 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f777113c9ad google::LogMessage::Fail() at ??:0
@ 0x7f777113f88a google::LogMessage::SendToLog() at ??:0
@ 0x7f777113c4de google::LogMessage::Flush() at ??:0
@ 0x7f777113da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7774ea07d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f7774e8f73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f7786b6b1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f7786b6a783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f7786bf62a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f7786bf60c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f7786c405e9 std::function<>::operator()() at ??:0
@ 0x7f7786b6ecdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f7786b6cc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f7786bf4610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f7786bf4439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f7773d88345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f7773d8b91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f7773d8b799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f7773d66eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f77770406db start_thread at ??:0
@ 0x7f776ffd661f clone at ??:0
W20260620 23:15:21.188040 27650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:21.188359 27650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:21.188529 27650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:21.195888 27650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:21.196041 27650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:21.196218 27650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.193
I20260620 23:15:21.205833 27650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:33627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.193:33111
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.191.193
--webserver_port=45075
--tserver_master_addrs=127.10.191.254:32873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.193
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:15:21.207149 27650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:21.208494 27650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:21.222330 27698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:21.222419 27695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:21.222347 27696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:21.223342 27650 server_base.cc:1034] running on GCE node
I20260620 23:15:21.223907 27650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:21.224853 27650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:21.226065 27650 hybrid_clock.cc:648] HybridClock initialized: now 1781997321226007 us; error 54 us; skew 500 ppm
I20260620 23:15:21.226400 27650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:15:21 dist-test-slave-jkbq krb5kdc[26500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997321, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:21.229693 27650 init.cc:377] Logged in from keytab as kudu/127.10.191.193@KRBTEST.COM (short username kudu)
I20260620 23:15:21.231259 27650 webserver.cc:460] Webserver started at http://127.10.191.193:45075/ using document root <none> and password file <none>
I20260620 23:15:21.231954 27650 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:21.232107 27650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:21.237563 27650 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 23:15:21.240703 27705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:21.242228 27650 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:15:21.242453 27650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/wal
uuid: "c5ff4a994ed24d10ae1b3b1f8fef3789"
format_stamp: "Formatted at 2026-06-20 23:14:35 on dist-test-slave-jkbq"
server_key: "eb9a2a15ac4996c235cb9c64bf327582"
server_key_iv: "85a36910358a037b75d27b2804e55630"
server_key_version: "encryptionkey@0"
I20260620 23:15:21.243568 27650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:21.263698 27650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:21.268251 27650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:21.268537 27650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:21.269613 27650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:15:21.271615 27712 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260620 23:15:21.277148 27712 ts_tablet_manager.cc:1852] T 05b2d6545c454f9a8a1a41fe8b78cb09 P c5ff4a994ed24d10ae1b3b1f8fef3789: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260620 23:15:21.278136 27712 ts_tablet_manager.cc:1905] T 05b2d6545c454f9a8a1a41fe8b78cb09 P c5ff4a994ed24d10ae1b3b1f8fef3789: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:15:21.286270 27712 ts_tablet_manager.cc:1918] T 05b2d6545c454f9a8a1a41fe8b78cb09 P c5ff4a994ed24d10ae1b3b1f8fef3789: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:15:21.286550 27712 ts_tablet_manager.cc:1939] T 05b2d6545c454f9a8a1a41fe8b78cb09 P c5ff4a994ed24d10ae1b3b1f8fef3789: Deleting consensus metadata
I20260620 23:15:21.287052 27650 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260620 23:15:21.287178 27650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s user 0.001s sys 0.000s
I20260620 23:15:21.287326 27650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:15:21.287426 27650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:21.322638 27650 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.193:33111
I20260620 23:15:21.322724 27818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.193:33111 every 8 connection(s)
I20260620 23:15:21.323982 27650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
I20260620 23:15:21.325516 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 27650
I20260620 23:15:21.326282 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.194:37623
--local_ip_for_outbound_sockets=127.10.191.194
--tserver_master_addrs=127.10.191.254:32873
--webserver_port=40087
--webserver_interface=127.10.191.194
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:33627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:15:21.329339 27821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:21.327368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.193:49157)
0620 23:15:21.328207 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:21.328468 (+ 261us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":352,"threads_started":1}
W20260620 23:15:21.330106 27819 heartbeater.cc:644] Failed to heartbeat to 127.10.191.254:32873 (0 consecutive failures): Network error: Failed to ping master at 127.10.191.254:32873: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
I20260620 23:15:21.331229 27821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:21.330734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.193:40661)
0620 23:15:21.330960 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:21.331064 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 23:15:21.332957 27821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:21.332570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.193:36927)
0620 23:15:21.332727 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:21.332823 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 23:15:21.333204 27819 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 23:15:21.647544 27822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:21.647855 27822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:21.648041 27822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:21.655194 27822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:21.655328 27822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:21.655488 27822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.194
I20260620 23:15:21.664856 27822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:33627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.194:37623
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.191.194
--webserver_port=40087
--tserver_master_addrs=127.10.191.254:32873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.194
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:15:21.666080 27822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:21.667435 27822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:21.680351 27827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:21.680418 27830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:21.680440 27828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:21.681180 27822 server_base.cc:1034] running on GCE node
I20260620 23:15:21.681746 27822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:21.682655 27822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:21.683897 27822 hybrid_clock.cc:648] HybridClock initialized: now 1781997321683845 us; error 74 us; skew 500 ppm
I20260620 23:15:21.684247 27822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:15:21 dist-test-slave-jkbq krb5kdc[26500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997321, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:21.687382 27822 init.cc:377] Logged in from keytab as kudu/127.10.191.194@KRBTEST.COM (short username kudu)
I20260620 23:15:21.688844 27822 webserver.cc:460] Webserver started at http://127.10.191.194:40087/ using document root <none> and password file <none>
I20260620 23:15:21.689507 27822 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:21.689652 27822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:21.695417 27822 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:15:21.699204 27837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:21.700888 27822 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 23:15:21.701047 27822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/wal
uuid: "30bbb7d7a0764e89aaf87e5609420583"
format_stamp: "Formatted at 2026-06-20 23:14:35 on dist-test-slave-jkbq"
server_key: "283a04a8e4f55dee6a1fe0c77108acf8"
server_key_iv: "931086b80c320db55475310070aa1715"
server_key_version: "encryptionkey@0"
I20260620 23:15:21.702229 27822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:21.722111 27822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:21.726289 27822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:21.726558 27822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:21.727519 27822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:15:21.728823 27822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:15:21.728960 27822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:21.729087 27822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:15:21.729173 27822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:21.762285 27822 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.194:37623
I20260620 23:15:21.762388 27949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.194:37623 every 8 connection(s)
I20260620 23:15:21.763796 27822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
I20260620 23:15:21.768610 27952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:21.766750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.194:59373)
0620 23:15:21.767519 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:21.767772 (+ 253us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":279,"threads_started":1}
W20260620 23:15:21.769222 27950 heartbeater.cc:644] Failed to heartbeat to 127.10.191.254:32873 (0 consecutive failures): Network error: Failed to ping master at 127.10.191.254:32873: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
I20260620 23:15:21.770081 27952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:21.769733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.194:58653)
0620 23:15:21.769911 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:21.769978 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 23:15:21.770161 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 27822
I20260620 23:15:21.770956 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.195:33237
--local_ip_for_outbound_sockets=127.10.191.195
--tserver_master_addrs=127.10.191.254:32873
--webserver_port=35375
--webserver_interface=127.10.191.195
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:33627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:15:21.771258 27952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:21.770776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.194:36299)
0620 23:15:21.770989 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:21.771105 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260620 23:15:21.771497 27950 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 23:15:22.094983 27953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:22.095323 27953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:22.095510 27953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:22.102738 27953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:22.102924 27953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:22.103121 27953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.195
I20260620 23:15:22.113180 27953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:33627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.195:33237
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.191.195
--webserver_port=35375
--tserver_master_addrs=127.10.191.254:32873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.195
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:15:22.114491 27953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:22.115844 27953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:22.128795 27958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:22.128825 27959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:22.128862 27961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:22.129762 27953 server_base.cc:1034] running on GCE node
I20260620 23:15:22.130386 27953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:22.131426 27953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:22.132659 27953 hybrid_clock.cc:648] HybridClock initialized: now 1781997322132619 us; error 52 us; skew 500 ppm
I20260620 23:15:22.132997 27953 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:15:22 dist-test-slave-jkbq krb5kdc[26500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997322, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:22.136188 27953 init.cc:377] Logged in from keytab as kudu/127.10.191.195@KRBTEST.COM (short username kudu)
I20260620 23:15:22.137666 27953 webserver.cc:460] Webserver started at http://127.10.191.195:35375/ using document root <none> and password file <none>
I20260620 23:15:22.138310 27953 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:22.138411 27953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:22.143821 27953 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:15:22.146479 27968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:22.148043 27953 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:15:22.148227 27953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/wal
uuid: "714cc984998f43f4b0ea18c70fa7382d"
format_stamp: "Formatted at 2026-06-20 23:14:36 on dist-test-slave-jkbq"
server_key: "2305fd1c25c65057b377af8592c2f6d4"
server_key_iv: "5a234915dde2c4e6e0084737c6e6d608"
server_key_version: "encryptionkey@0"
I20260620 23:15:22.149294 27953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:22.161517 27953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:22.165355 27953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:22.165582 27953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:22.166460 27953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:15:22.167753 27953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:15:22.167845 27953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:22.167928 27953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:15:22.167976 27953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:22.200904 27953 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.195:33237
I20260620 23:15:22.200981 28080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.195:33237 every 8 connection(s)
I20260620 23:15:22.202445 27953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
I20260620 23:15:22.202781 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 27953
I20260620 23:15:22.207938 28084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:22.205944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.195:34997)
0620 23:15:22.206807 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:22.207094 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":352,"threads_started":1}
W20260620 23:15:22.208608 28081 heartbeater.cc:644] Failed to heartbeat to 127.10.191.254:32873 (0 consecutive failures): Network error: Failed to ping master at 127.10.191.254:32873: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
I20260620 23:15:22.209513 28084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:22.209137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.195:37133)
0620 23:15:22.209330 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:22.209402 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260620 23:15:22.210374 28084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:22.210009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.195:47169)
0620 23:15:22.210146 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:22.210232 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 23:15:22.210605 28081 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 23:15:22.334595 28085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:22.333809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.193:46955)
0620 23:15:22.334278 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:22.334360 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":218,"threads_started":1}
I20260620 23:15:22.772830 28086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:22.772091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.194:56019)
0620 23:15:22.772526 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:22.772611 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":207,"threads_started":1}
I20260620 23:15:23.212077 28087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:23.211318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.195:56147)
0620 23:15:23.211751 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:23.211844 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":208,"threads_started":1}
I20260620 23:15:23.336194 28088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:23.335516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.193:32853)
0620 23:15:23.335949 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:23.336029 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":237,"threads_started":1}
I20260620 23:15:23.774645 28089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:23.773670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.194:38267)
0620 23:15:23.774319 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:23.774427 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":380,"threads_started":1}
I20260620 23:15:24.213850 28090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:24.213069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.195:43129)
0620 23:15:24.213597 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:24.213682 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":289,"threads_started":1}
I20260620 23:15:24.337837 28091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:24.337089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.193:54357)
0620 23:15:24.337571 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:24.337662 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":251,"threads_started":1}
I20260620 23:15:24.776356 28092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:24.775525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.194:47913)
0620 23:15:24.775998 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:24.776126 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":256,"threads_started":1}
I20260620 23:15:25.215976 28093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:25.215220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.195:34483)
0620 23:15:25.215717 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:25.215803 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":271,"threads_started":1}
I20260620 23:15:25.339502 28094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:25.338699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.193:48045)
0620 23:15:25.339218 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:25.339327 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":289,"threads_started":1}
I20260620 23:15:25.778004 28095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:25.777263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.194:60297)
0620 23:15:25.777720 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:25.777807 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":237,"threads_started":1}
I20260620 23:15:26.217774 28096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:26.217054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.195:44587)
0620 23:15:26.217497 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:26.217578 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":224,"threads_started":1}
I20260620 23:15:26.341112 28097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:26.340410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.193:59485)
0620 23:15:26.340871 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:26.340950 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":223,"threads_started":1}
I20260620 23:15:26.779798 28098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:26.779119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.194:50463)
0620 23:15:26.779556 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:26.779636 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":228,"threads_started":1}
I20260620 23:15:27.219589 28099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:27.218780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.195:40163)
0620 23:15:27.219334 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:27.219421 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":325,"threads_started":1}
I20260620 23:15:27.342821 28100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:27.342112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.193:47331)
0620 23:15:27.342562 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:27.342648 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":240,"threads_started":1}
I20260620 23:15:27.781335 28101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:27.780675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.194:54319)
0620 23:15:27.781087 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:27.781174 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":212,"threads_started":1}
I20260620 23:15:28.221164 28102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:28.220486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.195:38297)
0620 23:15:28.220916 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:28.221004 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":231,"threads_started":1}
I20260620 23:15:28.344532 28103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:28.343766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.193:33909)
0620 23:15:28.344252 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:28.344332 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":221,"threads_started":1}
I20260620 23:15:28.783104 28104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:28.782264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.194:43275)
0620 23:15:28.782746 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:28.782823 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":236,"threads_started":1}
I20260620 23:15:29.223176 28105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:29.222199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.195:35941)
0620 23:15:29.222804 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:29.222942 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":306,"threads_started":1}
I20260620 23:15:29.346065 28106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:29.345387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.193:47681)
0620 23:15:29.345807 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:29.345892 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":219,"threads_started":1}
I20260620 23:15:29.784667 28107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:29.783979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.194:46993)
0620 23:15:29.784430 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:29.784508 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":246,"threads_started":1}
I20260620 23:15:30.224781 28108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:30.224111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.195:35395)
0620 23:15:30.224548 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:30.224626 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":222,"threads_started":1}
I20260620 23:15:30.347569 28109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:30.346898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.193:55973)
0620 23:15:30.347337 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:30.347423 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":218,"threads_started":1}
I20260620 23:15:30.786366 28110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:30.785629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.194:50221)
0620 23:15:30.786096 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:30.786183 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":220,"threads_started":1}
I20260620 23:15:31.226608 28111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:31.225691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.195:57637)
0620 23:15:31.226246 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:31.226390 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":304,"threads_started":1}
I20260620 23:15:31.349037 28112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:31.348369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.193:51843)
0620 23:15:31.348781 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:31.348866 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":220,"threads_started":1}
I20260620 23:15:31.788293 28113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:31.787414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.194:35367)
0620 23:15:31.787972 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:31.788076 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":315,"threads_started":1}
I20260620 23:15:32.228358 28114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:32.227645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.195:60011)
0620 23:15:32.228118 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:32.228201 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":261,"threads_started":1}
I20260620 23:15:32.350811 28115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:32.350004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.193:50501)
0620 23:15:32.350558 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:32.350637 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":319,"threads_started":1}
I20260620 23:15:32.790124 28116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:32.789272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.194:38355)
0620 23:15:32.789846 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:32.789945 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":290,"threads_started":1}
I20260620 23:15:33.230032 28117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:33.229343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.195:49549)
0620 23:15:33.229769 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:33.229858 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":203,"threads_started":1}
I20260620 23:15:33.352735 28118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:33.351761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.193:56849)
0620 23:15:33.352360 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:33.352481 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":274,"threads_started":1}
I20260620 23:15:33.791985 28119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:33.791236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.194:58517)
0620 23:15:33.791731 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:33.791823 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":225,"threads_started":1}
I20260620 23:15:34.231786 28120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:34.231089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.195:47881)
0620 23:15:34.231536 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:34.231626 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":227,"threads_started":1}
I20260620 23:15:34.354673 28121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:34.353700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.193:58271)
0620 23:15:34.354296 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:34.354442 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":287,"threads_started":1}
I20260620 23:15:34.793663 28122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:34.792896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.194:36567)
0620 23:15:34.793377 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:34.793473 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":239,"threads_started":1}
I20260620 23:15:35.233523 28123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:35.232739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.195:43135)
0620 23:15:35.233220 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:35.233339 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":243,"threads_started":1}
I20260620 23:15:35.356349 28124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:35.355612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.193:35801)
0620 23:15:35.356058 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:35.356154 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":213,"threads_started":1}
I20260620 23:15:35.795526 28125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:35.794732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.194:43781)
0620 23:15:35.795228 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:35.795346 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":262,"threads_started":1}
I20260620 23:15:36.235414 28126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:36.234565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.195:51211)
0620 23:15:36.235124 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:36.235228 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":248,"threads_started":1}
I20260620 23:15:36.358268 28127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:36.357544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.193:33841)
0620 23:15:36.357995 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:36.358096 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":221,"threads_started":1}
I20260620 23:15:36.797322 28128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:36.796595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.194:34921)
0620 23:15:36.797028 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:36.797142 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":235,"threads_started":1}
I20260620 23:15:37.207599 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 27650
I20260620 23:15:37.230582 28129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:37.208714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47440 (local address 127.10.191.193:33111)
0620 23:15:37.209324 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:37.209356 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:37.209413 (+ 57us) server_negotiation.cc:407] Connection header received
0620 23:15:37.211429 (+ 2016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:37.211440 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:37.211882 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:37.212179 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:37.213587 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:37.214817 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:37.216107 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:37.216772 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:37.216912 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:37.224721 (+ 7809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:37.224729 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:37.224746 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:37.224785 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:37.227625 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:37.228278 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:37.228284 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:37.228291 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:37.228467 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:37.228771 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:37.228774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:37.228776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:37.229312 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:37.229549 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:37.229862 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 23:15:37.230283 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":284,"threads_started":1}
I20260620 23:15:37.232025 27814 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:15:37.237313 28131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:37.236504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:32873 (local address 127.10.191.195:44589)
0620 23:15:37.237038 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:37.237141 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.254:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":311,"threads_started":1}
I20260620 23:15:37.318046 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 27650
I20260620 23:15:37.335072 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 27822
I20260620 23:15:37.352577 28132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:37.335989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32800 (local address 127.10.191.194:37623)
0620 23:15:37.336508 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:37.336522 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:37.336559 (+ 37us) server_negotiation.cc:407] Connection header received
0620 23:15:37.338117 (+ 1558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:37.338126 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:37.338544 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:37.338935 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:37.340326 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:37.341539 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:37.342802 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:37.343503 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:37.343707 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:37.346323 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:37.346331 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:37.346345 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:37.346381 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:37.349135 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:37.349740 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:37.349747 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:37.349752 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:37.349928 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:37.350240 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:37.350243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:37.350245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:37.350724 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:37.351043 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:37.351498 (+ 455us) server_negotiation.cc:299] Negotiation successful
0620 23:15:37.352105 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":235,"threads_started":1}
I20260620 23:15:37.354075 27945 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:15:37.438392 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 27822
I20260620 23:15:37.455941 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 27953
I20260620 23:15:37.473217 28134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:37.456855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37782 (local address 127.10.191.195:33237)
0620 23:15:37.457462 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:37.457476 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:37.457517 (+ 41us) server_negotiation.cc:407] Connection header received
0620 23:15:37.459225 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:37.459234 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:37.459677 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:37.459954 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:37.461325 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:37.462517 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:37.463744 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:37.464451 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:37.464588 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:37.467113 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:37.467121 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:37.467136 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:37.467183 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:37.470084 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:37.470706 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:37.470712 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:37.470718 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:37.471124 (+ 406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:37.471455 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:37.471458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:37.471460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:37.471967 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:37.472202 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:37.472509 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 23:15:37.472925 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":285,"threads_started":1}
I20260620 23:15:37.474643 28076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:15:37.557727 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 27953
2026-06-20T23:15:37Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64506 ms)
[----------] 30 tests from SecurityITest (294629 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:15:37 dist-test-slave-jkbq krb5kdc[28138](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:15:37 dist-test-slave-jkbq krb5kdc[28138](info): set up 2 sockets
Jun 20 23:15:37 dist-test-slave-jkbq krb5kdc[28138](info): commencing operation
krb5kdc: starting...
W20260620 23:15:38.659165 11007 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Jun 20 23:15:38 dist-test-slave-jkbq krb5kdc[28138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:15:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:15:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.191.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
I20260620 23:15:38.894459 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:35755
--webserver_interface=127.10.191.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:39799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:35755
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:15:39.220427 28151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:39.220714 28151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:39.220894 28151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:39.228005 28151 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:15:39.228106 28151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:39.228192 28151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:39.228295 28151 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:15:39.228385 28151 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:15:39.237953 28151 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:39799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:35755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:35755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:15:39.239303 28151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:39.240617 28151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:39.252120 28157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:39.252213 28159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:39.252099 28156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:39.253350 28151 server_base.cc:1034] running on GCE node
I20260620 23:15:39.254109 28151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:39.255472 28151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:39.256671 28151 hybrid_clock.cc:648] HybridClock initialized: now 1781997339256626 us; error 51 us; skew 500 ppm
I20260620 23:15:39.257014 28151 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:15:39 dist-test-slave-jkbq krb5kdc[28138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997339, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:39.260093 28151 init.cc:377] Logged in from keytab as kudu/127.10.191.254@KRBTEST.COM (short username kudu)
I20260620 23:15:39.261539 28151 webserver.cc:460] Webserver started at http://127.10.191.254:35949/ using document root <none> and password file <none>
I20260620 23:15:39.262199 28151 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:39.262303 28151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:39.262544 28151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:15:39.265488 28151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "eafd1f04152d4c92969c13d1dbaff617"
format_stamp: "Formatted at 2026-06-20 23:15:39 on dist-test-slave-jkbq"
server_key: "03e75d01b22dbce3c1cb4f751b60d59e"
server_key_iv: "c1ee174fcbf7d5b5daf68ebfd0a00233"
server_key_version: "encryptionkey@0"
I20260620 23:15:39.266166 28151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "eafd1f04152d4c92969c13d1dbaff617"
format_stamp: "Formatted at 2026-06-20 23:15:39 on dist-test-slave-jkbq"
server_key: "03e75d01b22dbce3c1cb4f751b60d59e"
server_key_iv: "c1ee174fcbf7d5b5daf68ebfd0a00233"
server_key_version: "encryptionkey@0"
I20260620 23:15:39.271260 28151 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 23:15:39.275019 28166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:39.276360 28151 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:15:39.276544 28151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "eafd1f04152d4c92969c13d1dbaff617"
format_stamp: "Formatted at 2026-06-20 23:15:39 on dist-test-slave-jkbq"
server_key: "03e75d01b22dbce3c1cb4f751b60d59e"
server_key_iv: "c1ee174fcbf7d5b5daf68ebfd0a00233"
server_key_version: "encryptionkey@0"
I20260620 23:15:39.276733 28151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:39.292515 28151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:39.296538 28151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:39.296820 28151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:39.315373 28151 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:35755
I20260620 23:15:39.315393 28217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:35755 every 8 connection(s)
I20260620 23:15:39.316723 28151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:15:39.317812 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 28151
I20260620 23:15:39.318063 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:15:39.319002 11007 external_mini_cluster.cc:1428] Setting key 29cd772b980796c9ebe1655f314affb4
I20260620 23:15:39.321871 28218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 23:15:39 dist-test-slave-jkbq krb5kdc[28138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:15:39.333403 28218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eafd1f04152d4c92969c13d1dbaff617: Bootstrap starting.
I20260620 23:15:39.337013 28218 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eafd1f04152d4c92969c13d1dbaff617: Neither blocks nor log segments found. Creating new log.
I20260620 23:15:39.338068 28218 log.cc:824] T 00000000000000000000000000000000 P eafd1f04152d4c92969c13d1dbaff617: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:39.340756 28221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:39.321246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35424 (local address 127.10.191.254:35755)
0620 23:15:39.322092 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:39.322105 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:39.322152 (+ 47us) server_negotiation.cc:407] Connection header received
0620 23:15:39.323870 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:39.323893 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:39.324259 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:39.324649 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:39.326120 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:39.327523 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:39.328820 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:39.329477 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:39.329661 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:39.333743 (+ 4082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:39.333751 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:39.333765 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:39.333798 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:39.336780 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:39.337478 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:39.337485 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:39.337490 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:39.337643 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:39.338014 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:39.338020 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:39.338023 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:39.338597 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:39.338934 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:39.339408 (+ 474us) server_negotiation.cc:299] Negotiation successful
0620 23:15:39.339804 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":347,"threads_started":1}
I20260620 23:15:39.341167 28218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eafd1f04152d4c92969c13d1dbaff617: No bootstrap required, opened a new log
I20260620 23:15:39.349356 28218 raft_consensus.cc:348] T 00000000000000000000000000000000 P eafd1f04152d4c92969c13d1dbaff617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eafd1f04152d4c92969c13d1dbaff617" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 35755 } }
I20260620 23:15:39.349733 28218 raft_consensus.cc:374] T 00000000000000000000000000000000 P eafd1f04152d4c92969c13d1dbaff617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:15:39.349861 28218 raft_consensus.cc:729] T 00000000000000000000000000000000 P eafd1f04152d4c92969c13d1dbaff617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eafd1f04152d4c92969c13d1dbaff617, State: Initialized, Role: FOLLOWER
I20260620 23:15:39.350394 28218 consensus_queue.cc:260] T 00000000000000000000000000000000 P eafd1f04152d4c92969c13d1dbaff617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eafd1f04152d4c92969c13d1dbaff617" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 35755 } }
I20260620 23:15:39.350641 28218 raft_consensus.cc:388] T 00000000000000000000000000000000 P eafd1f04152d4c92969c13d1dbaff617 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:15:39.350795 28218 raft_consensus.cc:482] T 00000000000000000000000000000000 P eafd1f04152d4c92969c13d1dbaff617 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:15:39.351011 28218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eafd1f04152d4c92969c13d1dbaff617 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:15:39.353245 28218 raft_consensus.cc:504] T 00000000000000000000000000000000 P eafd1f04152d4c92969c13d1dbaff617 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eafd1f04152d4c92969c13d1dbaff617" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 35755 } }
I20260620 23:15:39.353744 28218 leader_election.cc:302] T 00000000000000000000000000000000 P eafd1f04152d4c92969c13d1dbaff617 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eafd1f04152d4c92969c13d1dbaff617; no voters:
I20260620 23:15:39.354290 28218 leader_election.cc:288] T 00000000000000000000000000000000 P eafd1f04152d4c92969c13d1dbaff617 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:15:39.354409 28223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eafd1f04152d4c92969c13d1dbaff617 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:15:39.355222 28223 raft_consensus.cc:686] T 00000000000000000000000000000000 P eafd1f04152d4c92969c13d1dbaff617 [term 1 LEADER]: Becoming Leader. State: Replica: eafd1f04152d4c92969c13d1dbaff617, State: Running, Role: LEADER
I20260620 23:15:39.355742 28223 consensus_queue.cc:237] T 00000000000000000000000000000000 P eafd1f04152d4c92969c13d1dbaff617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eafd1f04152d4c92969c13d1dbaff617" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 35755 } }
I20260620 23:15:39.356674 28218 sys_catalog.cc:564] T 00000000000000000000000000000000 P eafd1f04152d4c92969c13d1dbaff617 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:15:39.358747 28225 sys_catalog.cc:455] T 00000000000000000000000000000000 P eafd1f04152d4c92969c13d1dbaff617 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eafd1f04152d4c92969c13d1dbaff617. Latest consensus state: current_term: 1 leader_uuid: "eafd1f04152d4c92969c13d1dbaff617" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eafd1f04152d4c92969c13d1dbaff617" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 35755 } } }
I20260620 23:15:39.359292 28225 sys_catalog.cc:458] T 00000000000000000000000000000000 P eafd1f04152d4c92969c13d1dbaff617 [sys.catalog]: This master's current role is: LEADER
I20260620 23:15:39.358850 28224 sys_catalog.cc:455] T 00000000000000000000000000000000 P eafd1f04152d4c92969c13d1dbaff617 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eafd1f04152d4c92969c13d1dbaff617" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eafd1f04152d4c92969c13d1dbaff617" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 35755 } } }
I20260620 23:15:39.359622 28224 sys_catalog.cc:458] T 00000000000000000000000000000000 P eafd1f04152d4c92969c13d1dbaff617 [sys.catalog]: This master's current role is: LEADER
I20260620 23:15:39.360984 28231 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:15:39.366302 28231 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:15:39.376394 28231 catalog_manager.cc:1347] Generated new cluster ID: f0d6c7677f1844bfab5aa63e302c7456
I20260620 23:15:39.376509 28231 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:15:39.422588 28231 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:15:39.424501 28231 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:15:39.450003 28231 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eafd1f04152d4c92969c13d1dbaff617: Generated new TSK 0
I20260620 23:15:39.451084 28231 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.191.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab.
Entry for principal kudu/127.10.191.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab.
I20260620 23:15:39.518272 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.193:0
--local_ip_for_outbound_sockets=127.10.191.193
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:35755
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:39799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:15:39.842672 28244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:39.843019 28244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:39.843186 28244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:39.850337 28244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:39.850441 28244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:39.850637 28244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.193
I20260620 23:15:39.860653 28244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:39799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:35755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.193
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:15:39.862013 28244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:39.863382 28244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:39.875780 28249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:39.875831 28252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:39.875991 28250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:39.876582 28244 server_base.cc:1034] running on GCE node
I20260620 23:15:39.877135 28244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:39.878016 28244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:39.879247 28244 hybrid_clock.cc:648] HybridClock initialized: now 1781997339879174 us; error 54 us; skew 500 ppm
I20260620 23:15:39.879551 28244 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:15:39 dist-test-slave-jkbq krb5kdc[28138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997339, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:39.882948 28244 init.cc:377] Logged in from keytab as kudu/127.10.191.193@KRBTEST.COM (short username kudu)
I20260620 23:15:39.884398 28244 webserver.cc:460] Webserver started at http://127.10.191.193:41423/ using document root <none> and password file <none>
I20260620 23:15:39.885066 28244 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:39.885205 28244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:39.885514 28244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:15:39.888355 28244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-0/data/instance:
uuid: "3c908d6bca48408fb02bc59ffe8a5364"
format_stamp: "Formatted at 2026-06-20 23:15:39 on dist-test-slave-jkbq"
server_key: "4446a63f0195746654f5dfe19e48c143"
server_key_iv: "ad14a40b330f0e164a5395ea8af70169"
server_key_version: "encryptionkey@0"
I20260620 23:15:39.889160 28244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance:
uuid: "3c908d6bca48408fb02bc59ffe8a5364"
format_stamp: "Formatted at 2026-06-20 23:15:39 on dist-test-slave-jkbq"
server_key: "4446a63f0195746654f5dfe19e48c143"
server_key_iv: "ad14a40b330f0e164a5395ea8af70169"
server_key_version: "encryptionkey@0"
I20260620 23:15:39.894220 28244 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 23:15:39.897603 28259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:39.898921 28244 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:15:39.899081 28244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-0/wal
uuid: "3c908d6bca48408fb02bc59ffe8a5364"
format_stamp: "Formatted at 2026-06-20 23:15:39 on dist-test-slave-jkbq"
server_key: "4446a63f0195746654f5dfe19e48c143"
server_key_iv: "ad14a40b330f0e164a5395ea8af70169"
server_key_version: "encryptionkey@0"
I20260620 23:15:39.899281 28244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:39.909634 28244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:39.913460 28244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:39.913739 28244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:39.914630 28244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:15:39.915949 28244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:15:39.916044 28244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:39.916188 28244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:15:39.916260 28244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:39.949599 28244 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.193:42143
I20260620 23:15:39.949698 28371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.193:42143 every 8 connection(s)
I20260620 23:15:39.951079 28244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
I20260620 23:15:39.961428 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 28244
I20260620 23:15:39.961628 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance
I20260620 23:15:39.962460 11007 external_mini_cluster.cc:1428] Setting key 6e6c8c152bbf5e4c7edff5cbb462eb69
Jun 20 23:15:39 dist-test-slave-jkbq krb5kdc[28138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997339, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:15:39.971045 28374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:39.953709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:49139 (local address 127.10.191.254:35755)
0620 23:15:39.954130 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:39.954134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:39.955154 (+ 1020us) server_negotiation.cc:407] Connection header received
0620 23:15:39.957079 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:39.957084 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:39.957173 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:39.957287 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:39.958805 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:39.959570 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:39.961101 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:39.961590 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:39.961769 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:39.965542 (+ 3773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:39.965547 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:39.965550 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:39.965580 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:39.968096 (+ 2516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:39.968787 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:39.968793 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:39.968797 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:39.968921 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:39.969328 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:39.969333 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:39.969336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:39.969597 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:39.969749 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:39.970419 (+ 670us) server_negotiation.cc:299] Negotiation successful
0620 23:15:39.970637 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":204,"threads_started":1}
I20260620 23:15:39.971939 28375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:39.954291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:35755 (local address 127.10.191.193:49139)
0620 23:15:39.955008 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:39.955050 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:39.956805 (+ 1755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:39.957512 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:39.957519 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:39.958103 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:39.958646 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:39.958663 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:39.959718 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:39.959724 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:39.960926 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:39.960937 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:39.961882 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:39.961890 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:39.962214 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:39.962923 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:15:39.962955 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:39.965337 (+ 2382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:39.968233 (+ 2896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:39.968241 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:39.968256 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:39.968670 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:39.969024 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:39.969029 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:39.969033 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:39.969213 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:39.969727 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:39.969735 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:39.970051 (+ 316us) client_negotiation.cc:769] Sending connection context
0620 23:15:39.970309 (+ 258us) client_negotiation.cc:240] Negotiation successful
0620 23:15:39.970800 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":261,"threads_started":1}
I20260620 23:15:39.973451 28372 heartbeater.cc:347] Connected to a master server at 127.10.191.254:35755
I20260620 23:15:39.973742 28372 heartbeater.cc:464] Registering TS with master...
I20260620 23:15:39.974402 28372 heartbeater.cc:511] Master 127.10.191.254:35755 requested a full tablet report, sending...
I20260620 23:15:39.976635 28183 ts_manager.cc:194] Registered new tserver with Master: 3c908d6bca48408fb02bc59ffe8a5364 (127.10.191.193:42143)
I20260620 23:15:39.978787 28183 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.193@KRBTEST.COM'} at 127.10.191.193:49139
WARNING: no policy specified for kudu/127.10.191.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab.
Entry for principal kudu/127.10.191.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab.
I20260620 23:15:40.014210 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.194:0
--local_ip_for_outbound_sockets=127.10.191.194
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:35755
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:39799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:15:40.337576 28378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:40.337838 28378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:40.337929 28378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:40.345116 28378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:40.345219 28378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:40.345336 28378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.194
I20260620 23:15:40.355325 28378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:39799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:35755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.194
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:15:40.356465 28378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:40.358021 28378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:40.370920 28383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:40.370986 28386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:40.370963 28384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:40.371917 28378 server_base.cc:1034] running on GCE node
I20260620 23:15:40.372443 28378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:40.373306 28378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:40.374516 28378 hybrid_clock.cc:648] HybridClock initialized: now 1781997340374458 us; error 33 us; skew 500 ppm
I20260620 23:15:40.374930 28378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:15:40 dist-test-slave-jkbq krb5kdc[28138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997340, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:40.378798 28378 init.cc:377] Logged in from keytab as kudu/127.10.191.194@KRBTEST.COM (short username kudu)
I20260620 23:15:40.380254 28378 webserver.cc:460] Webserver started at http://127.10.191.194:40283/ using document root <none> and password file <none>
I20260620 23:15:40.380944 28378 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:40.381088 28378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:40.381408 28378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:15:40.384371 28378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-1/data/instance:
uuid: "6223daf297724094a6be178033a6e9ea"
format_stamp: "Formatted at 2026-06-20 23:15:40 on dist-test-slave-jkbq"
server_key: "72e0ec9095dda284f417ba01f9f58543"
server_key_iv: "baebcc7a028d6077ced2ede4628ff9fb"
server_key_version: "encryptionkey@0"
I20260620 23:15:40.385143 28378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance:
uuid: "6223daf297724094a6be178033a6e9ea"
format_stamp: "Formatted at 2026-06-20 23:15:40 on dist-test-slave-jkbq"
server_key: "72e0ec9095dda284f417ba01f9f58543"
server_key_iv: "baebcc7a028d6077ced2ede4628ff9fb"
server_key_version: "encryptionkey@0"
I20260620 23:15:40.390342 28378 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 23:15:40.393951 28393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:40.395304 28378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260620 23:15:40.395478 28378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-1/wal
uuid: "6223daf297724094a6be178033a6e9ea"
format_stamp: "Formatted at 2026-06-20 23:15:40 on dist-test-slave-jkbq"
server_key: "72e0ec9095dda284f417ba01f9f58543"
server_key_iv: "baebcc7a028d6077ced2ede4628ff9fb"
server_key_version: "encryptionkey@0"
I20260620 23:15:40.395678 28378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:40.414820 28378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:40.418771 28378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:40.419099 28378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:40.420059 28378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:15:40.421365 28378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:15:40.421461 28378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:40.421599 28378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:15:40.421669 28378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:40.454754 28378 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.194:33761
I20260620 23:15:40.454792 28505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.194:33761 every 8 connection(s)
I20260620 23:15:40.456073 28378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
I20260620 23:15:40.456391 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 28378
I20260620 23:15:40.456578 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance
I20260620 23:15:40.457207 11007 external_mini_cluster.cc:1428] Setting key 58cac6babff788aede3d902bd3dfaf69
Jun 20 23:15:40 dist-test-slave-jkbq krb5kdc[28138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997340, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:15:40.480543 28374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:40.458896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:47851 (local address 127.10.191.254:35755)
0620 23:15:40.459091 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:40.459098 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:40.460588 (+ 1490us) server_negotiation.cc:407] Connection header received
0620 23:15:40.463612 (+ 3024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:40.463617 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:40.463687 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:40.463770 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:40.465917 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:40.466626 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:40.469097 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:40.469851 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:40.470016 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:40.473554 (+ 3538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:40.473562 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:40.473570 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:40.473614 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:40.477282 (+ 3668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:40.477983 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:40.477991 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:40.477998 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:40.478151 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:40.478544 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:40.478552 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:40.478558 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:40.479020 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:40.479180 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:40.479818 (+ 638us) server_negotiation.cc:299] Negotiation successful
0620 23:15:40.480095 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 23:15:40.481544 28508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:40.459494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:35755 (local address 127.10.191.194:47851)
0620 23:15:40.460422 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:40.460480 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:40.463298 (+ 2818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:40.464048 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:40.464061 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:40.464876 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:40.465726 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:40.465754 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:40.466783 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:40.466791 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:40.468953 (+ 2162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:40.468965 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:40.470065 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:40.470069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:40.470264 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:40.471010 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:15:40.471054 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:40.473320 (+ 2266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:40.477409 (+ 4089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:40.477417 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:40.477429 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:40.477838 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:40.478253 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:40.478257 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:40.478259 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:40.478436 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:40.479132 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:40.479137 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:40.479447 (+ 310us) client_negotiation.cc:769] Sending connection context
0620 23:15:40.479670 (+ 223us) client_negotiation.cc:240] Negotiation successful
0620 23:15:40.480264 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":300,"threads_started":1}
I20260620 23:15:40.483075 28506 heartbeater.cc:347] Connected to a master server at 127.10.191.254:35755
I20260620 23:15:40.483412 28506 heartbeater.cc:464] Registering TS with master...
I20260620 23:15:40.484346 28506 heartbeater.cc:511] Master 127.10.191.254:35755 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.191.195@KRBTEST.COM; defaulting to no policy
I20260620 23:15:40.485971 28183 ts_manager.cc:194] Registered new tserver with Master: 6223daf297724094a6be178033a6e9ea (127.10.191.194:33761)
I20260620 23:15:40.486836 28183 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.194@KRBTEST.COM'} at 127.10.191.194:47851
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab.
Entry for principal kudu/127.10.191.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab.
I20260620 23:15:40.514619 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.195:0
--local_ip_for_outbound_sockets=127.10.191.195
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:35755
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:39799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:15:40.855712 28511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:40.856040 28511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:40.856202 28511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:40.863827 28511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:40.863979 28511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:40.864145 28511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.195
I20260620 23:15:40.873790 28511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:39799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:35755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.195
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:15:40.875185 28511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:40.876535 28511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:40.889492 28519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:40.889557 28517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:40.889521 28516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:40.890407 28511 server_base.cc:1034] running on GCE node
I20260620 23:15:40.891105 28511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:40.892033 28511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:40.893250 28511 hybrid_clock.cc:648] HybridClock initialized: now 1781997340893191 us; error 61 us; skew 500 ppm
I20260620 23:15:40.893599 28511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:15:40 dist-test-slave-jkbq krb5kdc[28138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997340, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:40.896728 28511 init.cc:377] Logged in from keytab as kudu/127.10.191.195@KRBTEST.COM (short username kudu)
I20260620 23:15:40.898137 28511 webserver.cc:460] Webserver started at http://127.10.191.195:35579/ using document root <none> and password file <none>
I20260620 23:15:40.898828 28511 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:40.898979 28511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:40.899264 28511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:15:40.902082 28511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-2/data/instance:
uuid: "43482947c07045c3bb1868081e82bda0"
format_stamp: "Formatted at 2026-06-20 23:15:40 on dist-test-slave-jkbq"
server_key: "5bcfb3ef1d5120358486a6b06a5186e5"
server_key_iv: "2feb429a94e882a1cf3b904835eac2ea"
server_key_version: "encryptionkey@0"
I20260620 23:15:40.903007 28511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance:
uuid: "43482947c07045c3bb1868081e82bda0"
format_stamp: "Formatted at 2026-06-20 23:15:40 on dist-test-slave-jkbq"
server_key: "5bcfb3ef1d5120358486a6b06a5186e5"
server_key_iv: "2feb429a94e882a1cf3b904835eac2ea"
server_key_version: "encryptionkey@0"
I20260620 23:15:40.908149 28511 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 23:15:40.911514 28526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:40.912748 28511 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:15:40.912914 28511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-2/wal
uuid: "43482947c07045c3bb1868081e82bda0"
format_stamp: "Formatted at 2026-06-20 23:15:40 on dist-test-slave-jkbq"
server_key: "5bcfb3ef1d5120358486a6b06a5186e5"
server_key_iv: "2feb429a94e882a1cf3b904835eac2ea"
server_key_version: "encryptionkey@0"
I20260620 23:15:40.913116 28511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:40.932715 28511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:40.937067 28511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:40.937348 28511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:40.938257 28511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:15:40.939568 28511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:15:40.939671 28511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:40.939812 28511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:15:40.939888 28511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:40.974726 28511 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.195:39807
I20260620 23:15:40.974783 28638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.195:39807 every 8 connection(s)
I20260620 23:15:40.976052 28511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
I20260620 23:15:40.977758 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 28511
I20260620 23:15:40.977941 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance
I20260620 23:15:40.978361 11007 external_mini_cluster.cc:1428] Setting key 71e599c5377b0a1faeac8c9a407baccf
I20260620 23:15:40.981788 28372 heartbeater.cc:503] Master 127.10.191.254:35755 was elected leader, sending a full tablet report...
Jun 20 23:15:40 dist-test-slave-jkbq krb5kdc[28138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997340, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:15:40.995877 28374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:40.978669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:57575 (local address 127.10.191.254:35755)
0620 23:15:40.978911 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:40.978917 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:40.980281 (+ 1364us) server_negotiation.cc:407] Connection header received
0620 23:15:40.982913 (+ 2632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:40.982919 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:40.982994 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:40.983070 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:40.984556 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:40.985286 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:40.986745 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:40.987208 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:40.987301 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:40.990525 (+ 3224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:40.990529 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:40.990532 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:40.990563 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:40.993029 (+ 2466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:40.993891 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:40.993896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:40.993898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:40.994044 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:40.994412 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:40.994416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:40.994417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:40.994700 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:40.994800 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:40.995479 (+ 679us) server_negotiation.cc:299] Negotiation successful
0620 23:15:40.995633 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 23:15:40.996824 28641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:40.979250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:35755 (local address 127.10.191.195:57575)
0620 23:15:40.980103 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:40.980164 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:40.982562 (+ 2398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:40.983334 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:40.983342 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:40.983848 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:40.984410 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:40.984428 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:40.985417 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:40.985421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:40.986627 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:40.986638 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:40.987411 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:40.987416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:40.987601 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:40.988277 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:15:40.988303 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:40.990347 (+ 2044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:40.993141 (+ 2794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:40.993149 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:40.993166 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:40.993768 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:40.994143 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:40.994146 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:40.994149 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:40.994317 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:40.994805 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:40.994811 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:40.995162 (+ 351us) client_negotiation.cc:769] Sending connection context
0620 23:15:40.995391 (+ 229us) client_negotiation.cc:240] Negotiation successful
0620 23:15:40.995739 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":311,"threads_started":1}
I20260620 23:15:40.998090 28639 heartbeater.cc:347] Connected to a master server at 127.10.191.254:35755
I20260620 23:15:40.998358 28639 heartbeater.cc:464] Registering TS with master...
I20260620 23:15:40.999122 28639 heartbeater.cc:511] Master 127.10.191.254:35755 requested a full tablet report, sending...
I20260620 23:15:41.000468 28183 ts_manager.cc:194] Registered new tserver with Master: 43482947c07045c3bb1868081e82bda0 (127.10.191.195:39807)
I20260620 23:15:41.001219 28183 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.195@KRBTEST.COM'} at 127.10.191.195:57575
I20260620 23:15:41.006067 11007 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:15:41.023214 28374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:41.010616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35436 (local address 127.10.191.254:35755)
0620 23:15:41.010796 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:41.010800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:41.011194 (+ 394us) server_negotiation.cc:407] Connection header received
0620 23:15:41.011327 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:41.011331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:41.011391 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:41.011499 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:41.012943 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:41.013695 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:41.014952 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:41.015354 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:41.015446 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:41.017775 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:41.017780 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:41.017783 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:41.017812 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:41.020461 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:41.021178 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:41.021188 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:41.021191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:41.021333 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:41.021704 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:41.021710 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:41.021713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:41.022059 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:41.022346 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:41.022644 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 23:15:41.022890 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 23:15:41.026904 28183 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:15:41.033602 28183 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:15:41.066282 28653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:41.046942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.194:33761 (local address 127.0.0.1:41228)
0620 23:15:41.057752 (+ 10810us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:41.057774 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:41.057872 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:41.059019 (+ 1147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:41.059024 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:41.059052 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:41.059687 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:41.059703 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:41.061301 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:41.061306 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:41.063738 (+ 2432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:41.063751 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:41.065308 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:41.065315 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:41.065508 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:41.065567 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 23:15:41.065705 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 23:15:41.065897 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10666,"mutex_wait_us":70,"thread_start_us":10152,"threads_started":1}
I20260620 23:15:41.066295 28656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:41.047207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41228 (local address 127.10.191.194:33761)
0620 23:15:41.058381 (+ 11174us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:41.058389 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:41.058418 (+ 29us) server_negotiation.cc:407] Connection header received
0620 23:15:41.058523 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:41.058533 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:41.058763 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:41.059475 (+ 712us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:41.059900 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:41.061127 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:41.063884 (+ 2757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:41.065100 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:41.065271 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:41.065465 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:41.065760 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 23:15:41.065938 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10991,"thread_start_us":10865,"threads_started":1}
I20260620 23:15:41.067018 28651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:41.044939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:42143 (local address 127.0.0.1:34284)
0620 23:15:41.045793 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:41.045853 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:41.046023 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:41.058808 (+ 12785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:41.058815 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:41.058853 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:41.059269 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:41.059278 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:41.061284 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:41.061294 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:41.064323 (+ 3029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:41.064354 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:41.066101 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:41.066109 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:41.066313 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:41.066354 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 23:15:41.066423 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 23:15:41.066576 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"mutex_wait_us":167,"thread_start_us":1042,"threads_started":1}
I20260620 23:15:41.067478 28655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:41.045593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.195:39807 (local address 127.0.0.1:42930)
0620 23:15:41.057405 (+ 11812us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:41.057429 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:41.057568 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:41.059317 (+ 1749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:41.059323 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:41.059347 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:41.059900 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:41.059912 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:41.061815 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:41.061819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:41.064374 (+ 2555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:41.064384 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:41.066589 (+ 2205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:41.066595 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:41.066772 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:41.066801 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 23:15:41.066942 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 23:15:41.067095 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11655,"mutex_wait_us":116,"thread_start_us":11841,"threads_started":1}
I20260620 23:15:41.067924 28654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:41.046152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42930 (local address 127.10.191.195:39807)
0620 23:15:41.058511 (+ 12359us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:41.058518 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:41.058546 (+ 28us) server_negotiation.cc:407] Connection header received
0620 23:15:41.058627 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:41.058633 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:41.058805 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:41.059019 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:41.060093 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:41.061609 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:41.064536 (+ 2927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:41.065604 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:41.065797 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:41.065968 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:41.067397 (+ 1429us) server_negotiation.cc:299] Negotiation successful
0620 23:15:41.067565 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12181,"thread_start_us":12177,"threads_started":1}
I20260620 23:15:41.068249 28652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:41.045255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34284 (local address 127.10.191.193:42143)
0620 23:15:41.058156 (+ 12901us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:41.058166 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:41.058196 (+ 30us) server_negotiation.cc:407] Connection header received
0620 23:15:41.058299 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:41.058310 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:41.058592 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:41.058802 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:41.059447 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:41.061133 (+ 1686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:41.064502 (+ 3369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:41.065862 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:41.067498 (+ 1636us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:41.067682 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:41.067752 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 23:15:41.067936 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12685,"thread_start_us":12539,"threads_started":1}
I20260620 23:15:41.070627 28441 tablet_service.cc:1467] Processing CreateTablet for tablet 683e394187ae4d23829257ce992c7914 (DEFAULT_TABLE table=test-table [id=1714c07e066a4e03acb82e350a69aa89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:15:41.071054 28574 tablet_service.cc:1467] Processing CreateTablet for tablet 683e394187ae4d23829257ce992c7914 (DEFAULT_TABLE table=test-table [id=1714c07e066a4e03acb82e350a69aa89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:15:41.072186 28441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 683e394187ae4d23829257ce992c7914. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:15:41.072247 28574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 683e394187ae4d23829257ce992c7914. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:15:41.071820 28307 tablet_service.cc:1467] Processing CreateTablet for tablet 683e394187ae4d23829257ce992c7914 (DEFAULT_TABLE table=test-table [id=1714c07e066a4e03acb82e350a69aa89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:15:41.073105 28307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 683e394187ae4d23829257ce992c7914. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:15:41.084908 28657 tablet_bootstrap.cc:492] T 683e394187ae4d23829257ce992c7914 P 43482947c07045c3bb1868081e82bda0: Bootstrap starting.
I20260620 23:15:41.087590 28658 tablet_bootstrap.cc:492] T 683e394187ae4d23829257ce992c7914 P 6223daf297724094a6be178033a6e9ea: Bootstrap starting.
I20260620 23:15:41.088271 28657 tablet_bootstrap.cc:654] T 683e394187ae4d23829257ce992c7914 P 43482947c07045c3bb1868081e82bda0: Neither blocks nor log segments found. Creating new log.
I20260620 23:15:41.089229 28659 tablet_bootstrap.cc:492] T 683e394187ae4d23829257ce992c7914 P 3c908d6bca48408fb02bc59ffe8a5364: Bootstrap starting.
I20260620 23:15:41.089629 28657 log.cc:824] T 683e394187ae4d23829257ce992c7914 P 43482947c07045c3bb1868081e82bda0: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:41.091128 28658 tablet_bootstrap.cc:654] T 683e394187ae4d23829257ce992c7914 P 6223daf297724094a6be178033a6e9ea: Neither blocks nor log segments found. Creating new log.
I20260620 23:15:41.092345 28658 log.cc:824] T 683e394187ae4d23829257ce992c7914 P 6223daf297724094a6be178033a6e9ea: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:41.092406 28659 tablet_bootstrap.cc:654] T 683e394187ae4d23829257ce992c7914 P 3c908d6bca48408fb02bc59ffe8a5364: Neither blocks nor log segments found. Creating new log.
I20260620 23:15:41.093020 28657 tablet_bootstrap.cc:492] T 683e394187ae4d23829257ce992c7914 P 43482947c07045c3bb1868081e82bda0: No bootstrap required, opened a new log
I20260620 23:15:41.093277 28657 ts_tablet_manager.cc:1397] T 683e394187ae4d23829257ce992c7914 P 43482947c07045c3bb1868081e82bda0: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260620 23:15:41.093567 28659 log.cc:824] T 683e394187ae4d23829257ce992c7914 P 3c908d6bca48408fb02bc59ffe8a5364: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:41.095372 28658 tablet_bootstrap.cc:492] T 683e394187ae4d23829257ce992c7914 P 6223daf297724094a6be178033a6e9ea: No bootstrap required, opened a new log
I20260620 23:15:41.095629 28658 ts_tablet_manager.cc:1397] T 683e394187ae4d23829257ce992c7914 P 6223daf297724094a6be178033a6e9ea: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260620 23:15:41.096326 28659 tablet_bootstrap.cc:492] T 683e394187ae4d23829257ce992c7914 P 3c908d6bca48408fb02bc59ffe8a5364: No bootstrap required, opened a new log
I20260620 23:15:41.096572 28659 ts_tablet_manager.cc:1397] T 683e394187ae4d23829257ce992c7914 P 3c908d6bca48408fb02bc59ffe8a5364: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 23:15:41.104223 28659 raft_consensus.cc:348] T 683e394187ae4d23829257ce992c7914 P 3c908d6bca48408fb02bc59ffe8a5364 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c908d6bca48408fb02bc59ffe8a5364" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 42143 } } peers { permanent_uuid: "6223daf297724094a6be178033a6e9ea" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 33761 } } peers { permanent_uuid: "43482947c07045c3bb1868081e82bda0" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39807 } }
I20260620 23:15:41.104607 28659 raft_consensus.cc:374] T 683e394187ae4d23829257ce992c7914 P 3c908d6bca48408fb02bc59ffe8a5364 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:15:41.104722 28659 raft_consensus.cc:729] T 683e394187ae4d23829257ce992c7914 P 3c908d6bca48408fb02bc59ffe8a5364 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c908d6bca48408fb02bc59ffe8a5364, State: Initialized, Role: FOLLOWER
I20260620 23:15:41.104512 28657 raft_consensus.cc:348] T 683e394187ae4d23829257ce992c7914 P 43482947c07045c3bb1868081e82bda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c908d6bca48408fb02bc59ffe8a5364" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 42143 } } peers { permanent_uuid: "6223daf297724094a6be178033a6e9ea" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 33761 } } peers { permanent_uuid: "43482947c07045c3bb1868081e82bda0" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39807 } }
I20260620 23:15:41.104964 28657 raft_consensus.cc:374] T 683e394187ae4d23829257ce992c7914 P 43482947c07045c3bb1868081e82bda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:15:41.105094 28657 raft_consensus.cc:729] T 683e394187ae4d23829257ce992c7914 P 43482947c07045c3bb1868081e82bda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43482947c07045c3bb1868081e82bda0, State: Initialized, Role: FOLLOWER
I20260620 23:15:41.105408 28659 consensus_queue.cc:260] T 683e394187ae4d23829257ce992c7914 P 3c908d6bca48408fb02bc59ffe8a5364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c908d6bca48408fb02bc59ffe8a5364" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 42143 } } peers { permanent_uuid: "6223daf297724094a6be178033a6e9ea" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 33761 } } peers { permanent_uuid: "43482947c07045c3bb1868081e82bda0" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39807 } }
I20260620 23:15:41.105692 28657 consensus_queue.cc:260] T 683e394187ae4d23829257ce992c7914 P 43482947c07045c3bb1868081e82bda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c908d6bca48408fb02bc59ffe8a5364" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 42143 } } peers { permanent_uuid: "6223daf297724094a6be178033a6e9ea" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 33761 } } peers { permanent_uuid: "43482947c07045c3bb1868081e82bda0" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39807 } }
I20260620 23:15:41.106348 28658 raft_consensus.cc:348] T 683e394187ae4d23829257ce992c7914 P 6223daf297724094a6be178033a6e9ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c908d6bca48408fb02bc59ffe8a5364" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 42143 } } peers { permanent_uuid: "6223daf297724094a6be178033a6e9ea" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 33761 } } peers { permanent_uuid: "43482947c07045c3bb1868081e82bda0" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39807 } }
I20260620 23:15:41.106940 28658 raft_consensus.cc:374] T 683e394187ae4d23829257ce992c7914 P 6223daf297724094a6be178033a6e9ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:15:41.107105 28658 raft_consensus.cc:729] T 683e394187ae4d23829257ce992c7914 P 6223daf297724094a6be178033a6e9ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6223daf297724094a6be178033a6e9ea, State: Initialized, Role: FOLLOWER
I20260620 23:15:41.107259 28659 ts_tablet_manager.cc:1428] T 683e394187ae4d23829257ce992c7914 P 3c908d6bca48408fb02bc59ffe8a5364: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260620 23:15:41.107837 28658 consensus_queue.cc:260] T 683e394187ae4d23829257ce992c7914 P 6223daf297724094a6be178033a6e9ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c908d6bca48408fb02bc59ffe8a5364" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 42143 } } peers { permanent_uuid: "6223daf297724094a6be178033a6e9ea" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 33761 } } peers { permanent_uuid: "43482947c07045c3bb1868081e82bda0" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39807 } }
I20260620 23:15:41.111423 28506 heartbeater.cc:503] Master 127.10.191.254:35755 was elected leader, sending a full tablet report...
I20260620 23:15:41.112309 28658 ts_tablet_manager.cc:1428] T 683e394187ae4d23829257ce992c7914 P 6223daf297724094a6be178033a6e9ea: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20260620 23:15:41.113090 28639 heartbeater.cc:503] Master 127.10.191.254:35755 was elected leader, sending a full tablet report...
I20260620 23:15:41.113994 28657 ts_tablet_manager.cc:1428] T 683e394187ae4d23829257ce992c7914 P 43482947c07045c3bb1868081e82bda0: Time spent starting tablet: real 0.021s user 0.012s sys 0.009s
I20260620 23:15:41.165890 28665 raft_consensus.cc:482] T 683e394187ae4d23829257ce992c7914 P 6223daf297724094a6be178033a6e9ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:15:41.166234 28665 raft_consensus.cc:504] T 683e394187ae4d23829257ce992c7914 P 6223daf297724094a6be178033a6e9ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c908d6bca48408fb02bc59ffe8a5364" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 42143 } } peers { permanent_uuid: "6223daf297724094a6be178033a6e9ea" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 33761 } } peers { permanent_uuid: "43482947c07045c3bb1868081e82bda0" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39807 } }
I20260620 23:15:41.168085 28665 leader_election.cc:288] T 683e394187ae4d23829257ce992c7914 P 6223daf297724094a6be178033a6e9ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c908d6bca48408fb02bc59ffe8a5364 (127.10.191.193:42143), 43482947c07045c3bb1868081e82bda0 (127.10.191.195:39807)
I20260620 23:15:41.175159 28654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:41.168537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:57537 (local address 127.10.191.195:39807)
0620 23:15:41.168772 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:41.168779 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:41.169731 (+ 952us) server_negotiation.cc:407] Connection header received
0620 23:15:41.169893 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:41.169898 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:41.169968 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:41.170047 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:41.170744 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:41.171543 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:41.173409 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:41.174198 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:41.174365 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:41.174457 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:41.174731 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 23:15:41.174861 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 23:15:41.175169 28652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:41.168538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:35015 (local address 127.10.191.193:42143)
0620 23:15:41.168782 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:41.168789 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:41.169552 (+ 763us) server_negotiation.cc:407] Connection header received
0620 23:15:41.169870 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:41.169875 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:41.169954 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:41.170043 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:41.170932 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:41.171653 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:41.173457 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:41.174214 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:41.174409 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:41.174481 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:41.174767 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 23:15:41.174916 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 23:15:41.175338 28666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:41.168569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:42143 (local address 127.10.191.194:35015)
0620 23:15:41.169429 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:41.169452 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:41.169613 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:41.170219 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:41.170226 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:41.170254 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:41.170745 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:41.170757 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:41.171794 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:41.171797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:41.173286 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:41.173297 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:41.174457 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:41.174463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:41.174603 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:41.174625 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 23:15:41.174729 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 23:15:41.174858 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"mutex_wait_us":43,"thread_start_us":464,"threads_started":1}
I20260620 23:15:41.175364 28667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:41.168574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.195:39807 (local address 127.10.191.194:57537)
0620 23:15:41.169601 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:41.169620 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:41.169755 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:41.170089 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:41.170093 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:41.170119 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:41.170563 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:41.170574 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:41.171669 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:41.171673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:41.173249 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:41.173264 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:41.174425 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:41.174431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:41.174582 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:41.174606 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 23:15:41.174680 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 23:15:41.174828 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":912,"mutex_wait_us":87,"thread_start_us":608,"threads_started":1}
I20260620 23:15:41.176530 28594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "683e394187ae4d23829257ce992c7914" candidate_uuid: "6223daf297724094a6be178033a6e9ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43482947c07045c3bb1868081e82bda0" is_pre_election: true
I20260620 23:15:41.176535 28327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "683e394187ae4d23829257ce992c7914" candidate_uuid: "6223daf297724094a6be178033a6e9ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c908d6bca48408fb02bc59ffe8a5364" is_pre_election: true
I20260620 23:15:41.177117 28327 raft_consensus.cc:2454] T 683e394187ae4d23829257ce992c7914 P 3c908d6bca48408fb02bc59ffe8a5364 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6223daf297724094a6be178033a6e9ea in term 0.
I20260620 23:15:41.177117 28594 raft_consensus.cc:2454] T 683e394187ae4d23829257ce992c7914 P 43482947c07045c3bb1868081e82bda0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6223daf297724094a6be178033a6e9ea in term 0.
I20260620 23:15:41.177868 28394 leader_election.cc:302] T 683e394187ae4d23829257ce992c7914 P 6223daf297724094a6be178033a6e9ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c908d6bca48408fb02bc59ffe8a5364, 6223daf297724094a6be178033a6e9ea; no voters:
I20260620 23:15:41.178320 28665 raft_consensus.cc:2781] T 683e394187ae4d23829257ce992c7914 P 6223daf297724094a6be178033a6e9ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:15:41.178478 28665 raft_consensus.cc:482] T 683e394187ae4d23829257ce992c7914 P 6223daf297724094a6be178033a6e9ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:15:41.178606 28665 raft_consensus.cc:3037] T 683e394187ae4d23829257ce992c7914 P 6223daf297724094a6be178033a6e9ea [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:15:41.180964 28665 raft_consensus.cc:504] T 683e394187ae4d23829257ce992c7914 P 6223daf297724094a6be178033a6e9ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c908d6bca48408fb02bc59ffe8a5364" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 42143 } } peers { permanent_uuid: "6223daf297724094a6be178033a6e9ea" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 33761 } } peers { permanent_uuid: "43482947c07045c3bb1868081e82bda0" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39807 } }
I20260620 23:15:41.181576 28665 leader_election.cc:288] T 683e394187ae4d23829257ce992c7914 P 6223daf297724094a6be178033a6e9ea [CANDIDATE]: Term 1 election: Requested vote from peers 3c908d6bca48408fb02bc59ffe8a5364 (127.10.191.193:42143), 43482947c07045c3bb1868081e82bda0 (127.10.191.195:39807)
I20260620 23:15:41.182031 28327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "683e394187ae4d23829257ce992c7914" candidate_uuid: "6223daf297724094a6be178033a6e9ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c908d6bca48408fb02bc59ffe8a5364"
I20260620 23:15:41.182042 28594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "683e394187ae4d23829257ce992c7914" candidate_uuid: "6223daf297724094a6be178033a6e9ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43482947c07045c3bb1868081e82bda0"
I20260620 23:15:41.182241 28327 raft_consensus.cc:3037] T 683e394187ae4d23829257ce992c7914 P 3c908d6bca48408fb02bc59ffe8a5364 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:15:41.182266 28594 raft_consensus.cc:3037] T 683e394187ae4d23829257ce992c7914 P 43482947c07045c3bb1868081e82bda0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:15:41.184477 28327 raft_consensus.cc:2454] T 683e394187ae4d23829257ce992c7914 P 3c908d6bca48408fb02bc59ffe8a5364 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6223daf297724094a6be178033a6e9ea in term 1.
I20260620 23:15:41.184485 28594 raft_consensus.cc:2454] T 683e394187ae4d23829257ce992c7914 P 43482947c07045c3bb1868081e82bda0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6223daf297724094a6be178033a6e9ea in term 1.
I20260620 23:15:41.184890 28394 leader_election.cc:302] T 683e394187ae4d23829257ce992c7914 P 6223daf297724094a6be178033a6e9ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c908d6bca48408fb02bc59ffe8a5364, 6223daf297724094a6be178033a6e9ea; no voters:
I20260620 23:15:41.185217 28665 raft_consensus.cc:2781] T 683e394187ae4d23829257ce992c7914 P 6223daf297724094a6be178033a6e9ea [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:15:41.185658 28665 raft_consensus.cc:686] T 683e394187ae4d23829257ce992c7914 P 6223daf297724094a6be178033a6e9ea [term 1 LEADER]: Becoming Leader. State: Replica: 6223daf297724094a6be178033a6e9ea, State: Running, Role: LEADER
I20260620 23:15:41.186210 28665 consensus_queue.cc:237] T 683e394187ae4d23829257ce992c7914 P 6223daf297724094a6be178033a6e9ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c908d6bca48408fb02bc59ffe8a5364" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 42143 } } peers { permanent_uuid: "6223daf297724094a6be178033a6e9ea" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 33761 } } peers { permanent_uuid: "43482947c07045c3bb1868081e82bda0" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39807 } }
I20260620 23:15:41.190730 28182 catalog_manager.cc:5495] T 683e394187ae4d23829257ce992c7914 P 6223daf297724094a6be178033a6e9ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 6223daf297724094a6be178033a6e9ea (127.10.191.194). New cstate: current_term: 1 leader_uuid: "6223daf297724094a6be178033a6e9ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c908d6bca48408fb02bc59ffe8a5364" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 42143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6223daf297724094a6be178033a6e9ea" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 33761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43482947c07045c3bb1868081e82bda0" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39807 } health_report { overall_health: UNKNOWN } } }
W20260620 23:15:41.203356 28373 tablet.cc:2369] T 683e394187ae4d23829257ce992c7914 P 3c908d6bca48408fb02bc59ffe8a5364: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:15:41.208184 28507 tablet.cc:2369] T 683e394187ae4d23829257ce992c7914 P 6223daf297724094a6be178033a6e9ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:15:41.211694 28656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:41.206285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41234 (local address 127.10.191.194:33761)
0620 23:15:41.206466 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:41.206470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:41.206612 (+ 142us) server_negotiation.cc:407] Connection header received
0620 23:15:41.206768 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:41.206772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:41.206856 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:41.207035 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:15:41.207357 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:41.208215 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:41.209718 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:41.210109 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:41.210203 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:41.210638 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:15:41.211187 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:15:41.211295 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:41.211425 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 23:15:41.211487 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:15:41.219774 28594 raft_consensus.cc:1261] T 683e394187ae4d23829257ce992c7914 P 43482947c07045c3bb1868081e82bda0 [term 1 FOLLOWER]: Refusing update from remote peer 6223daf297724094a6be178033a6e9ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:15:41.220773 28327 raft_consensus.cc:1261] T 683e394187ae4d23829257ce992c7914 P 3c908d6bca48408fb02bc59ffe8a5364 [term 1 FOLLOWER]: Refusing update from remote peer 6223daf297724094a6be178033a6e9ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:15:41.220837 28668 consensus_queue.cc:1035] T 683e394187ae4d23829257ce992c7914 P 6223daf297724094a6be178033a6e9ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "43482947c07045c3bb1868081e82bda0" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 39807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:15:41.221590 28668 consensus_queue.cc:1035] T 683e394187ae4d23829257ce992c7914 P 6223daf297724094a6be178033a6e9ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c908d6bca48408fb02bc59ffe8a5364" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 42143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 23:15:41.227648 28640 tablet.cc:2369] T 683e394187ae4d23829257ce992c7914 P 43482947c07045c3bb1868081e82bda0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:15:41.232896 28673 mvcc.cc:205] Tried to move back new op lower bound from 7299061109626236928 to 7299061109502783488. Current Snapshot: MvccSnapshot[applied={T|T < 7299061109626236928}]
I20260620 23:15:41.233104 28671 mvcc.cc:205] Tried to move back new op lower bound from 7299061109626236928 to 7299061109502783488. Current Snapshot: MvccSnapshot[applied={T|T < 7299061109626236928}]
I20260620 23:15:41.246189 28183 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:15:41.246479 28183 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:15:41.250020 28183 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eafd1f04152d4c92969c13d1dbaff617: Sending DeleteTablet for 3 replicas of tablet 683e394187ae4d23829257ce992c7914
I20260620 23:15:41.251362 28574 tablet_service.cc:1514] Processing DeleteTablet for tablet 683e394187ae4d23829257ce992c7914 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:15:41 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:42930
I20260620 23:15:41.251361 28307 tablet_service.cc:1514] Processing DeleteTablet for tablet 683e394187ae4d23829257ce992c7914 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:15:41 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:34284
I20260620 23:15:41.251502 28441 tablet_service.cc:1514] Processing DeleteTablet for tablet 683e394187ae4d23829257ce992c7914 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:15:41 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:41228
I20260620 23:15:41.253521 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 28244
I20260620 23:15:41.264400 28682 tablet_replica.cc:330] T 683e394187ae4d23829257ce992c7914 P 3c908d6bca48408fb02bc59ffe8a5364: stopping tablet replica
I20260620 23:15:41.264671 28681 tablet_replica.cc:330] T 683e394187ae4d23829257ce992c7914 P 43482947c07045c3bb1868081e82bda0: stopping tablet replica
I20260620 23:15:41.265296 28682 raft_consensus.cc:2229] T 683e394187ae4d23829257ce992c7914 P 3c908d6bca48408fb02bc59ffe8a5364 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:15:41.265936 28681 raft_consensus.cc:2229] T 683e394187ae4d23829257ce992c7914 P 43482947c07045c3bb1868081e82bda0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:15:41.266641 28681 raft_consensus.cc:2258] T 683e394187ae4d23829257ce992c7914 P 43482947c07045c3bb1868081e82bda0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:15:41.267259 28682 raft_consensus.cc:2258] T 683e394187ae4d23829257ce992c7914 P 3c908d6bca48408fb02bc59ffe8a5364 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:15:41.269797 28681 ts_tablet_manager.cc:1905] T 683e394187ae4d23829257ce992c7914 P 43482947c07045c3bb1868081e82bda0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:15:41.270197 28682 ts_tablet_manager.cc:1905] T 683e394187ae4d23829257ce992c7914 P 3c908d6bca48408fb02bc59ffe8a5364: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:15:41.274631 28683 tablet_replica.cc:330] T 683e394187ae4d23829257ce992c7914 P 6223daf297724094a6be178033a6e9ea: stopping tablet replica
I20260620 23:15:41.275113 28683 raft_consensus.cc:2229] T 683e394187ae4d23829257ce992c7914 P 6223daf297724094a6be178033a6e9ea [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:15:41.276862 28681 ts_tablet_manager.cc:1918] T 683e394187ae4d23829257ce992c7914 P 43482947c07045c3bb1868081e82bda0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:15:41.277179 28681 log.cc:1196] T 683e394187ae4d23829257ce992c7914 P 43482947c07045c3bb1868081e82bda0: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-2/wal/wals/683e394187ae4d23829257ce992c7914
I20260620 23:15:41.277642 28682 ts_tablet_manager.cc:1918] T 683e394187ae4d23829257ce992c7914 P 3c908d6bca48408fb02bc59ffe8a5364: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:15:41.277621 28683 raft_consensus.cc:2258] T 683e394187ae4d23829257ce992c7914 P 6223daf297724094a6be178033a6e9ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:15:41.277815 28681 ts_tablet_manager.cc:1939] T 683e394187ae4d23829257ce992c7914 P 43482947c07045c3bb1868081e82bda0: Deleting consensus metadata
I20260620 23:15:41.277905 28682 log.cc:1196] T 683e394187ae4d23829257ce992c7914 P 3c908d6bca48408fb02bc59ffe8a5364: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-0/wal/wals/683e394187ae4d23829257ce992c7914
I20260620 23:15:41.278527 28682 ts_tablet_manager.cc:1939] T 683e394187ae4d23829257ce992c7914 P 3c908d6bca48408fb02bc59ffe8a5364: Deleting consensus metadata
I20260620 23:15:41.280134 28170 catalog_manager.cc:4841] TS 43482947c07045c3bb1868081e82bda0 (127.10.191.195:39807): tablet 683e394187ae4d23829257ce992c7914 (table test-table [id=1714c07e066a4e03acb82e350a69aa89]) successfully deleted
I20260620 23:15:41.280443 28167 catalog_manager.cc:4841] TS 3c908d6bca48408fb02bc59ffe8a5364 (127.10.191.193:42143): tablet 683e394187ae4d23829257ce992c7914 (table test-table [id=1714c07e066a4e03acb82e350a69aa89]) successfully deleted
I20260620 23:15:41.282128 28683 ts_tablet_manager.cc:1905] T 683e394187ae4d23829257ce992c7914 P 6223daf297724094a6be178033a6e9ea: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 23:15:41 dist-test-slave-jkbq krb5kdc[28138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.193@KRBTEST.COM
I20260620 23:15:41.288790 28683 ts_tablet_manager.cc:1918] T 683e394187ae4d23829257ce992c7914 P 6223daf297724094a6be178033a6e9ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:15:41.288975 28683 log.cc:1196] T 683e394187ae4d23829257ce992c7914 P 6223daf297724094a6be178033a6e9ea: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781997042893763-11007-0/minicluster-data/ts-1/wal/wals/683e394187ae4d23829257ce992c7914
I20260620 23:15:41.289499 28683 ts_tablet_manager.cc:1939] T 683e394187ae4d23829257ce992c7914 P 6223daf297724094a6be178033a6e9ea: Deleting consensus metadata
I20260620 23:15:41.291188 28167 catalog_manager.cc:4841] TS 6223daf297724094a6be178033a6e9ea (127.10.191.194:33761): tablet 683e394187ae4d23829257ce992c7914 (table test-table [id=1714c07e066a4e03acb82e350a69aa89]) successfully deleted
I20260620 23:15:41.292213 28652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:41.255216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34292 (local address 127.10.191.193:42143)
0620 23:15:41.255393 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:41.255400 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:41.268512 (+ 13112us) server_negotiation.cc:407] Connection header received
0620 23:15:41.268729 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:41.268734 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:41.268825 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:41.268922 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:41.270577 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:41.271743 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:41.275361 (+ 3618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:41.276053 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:41.276169 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:41.285171 (+ 9002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:41.285185 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:41.285205 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:41.285253 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:41.289165 (+ 3912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:41.290040 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:41.290048 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:41.290051 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:41.290218 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:41.290667 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:41.290672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:41.290675 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:41.291168 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:41.291483 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:41.291758 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 23:15:41.291919 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 23:15:41.292968 28367 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:15:41.398366 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 28244
I20260620 23:15:41.421804 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 28378
Jun 20 23:15:41 dist-test-slave-jkbq krb5kdc[28138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.194@KRBTEST.COM
I20260620 23:15:41.438284 28656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:41.422620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41250 (local address 127.10.191.194:33761)
0620 23:15:41.422817 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:41.422823 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:41.422847 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:15:41.423047 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:41.423053 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:41.423183 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:41.423296 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:41.424511 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:41.425739 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:41.427131 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:41.427806 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:41.427933 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:41.432448 (+ 4515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:41.432464 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:41.432479 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:41.432518 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:41.435579 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:41.436400 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:41.436407 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:41.436409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:41.436543 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:41.436919 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:41.436923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:41.436925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:41.437267 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:41.437536 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:41.437816 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 23:15:41.437971 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 23:15:41.439116 28501 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:15:41.551811 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 28378
I20260620 23:15:41.570969 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 28511
Jun 20 23:15:41 dist-test-slave-jkbq krb5kdc[28138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.195@KRBTEST.COM
I20260620 23:15:41.585564 28654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:41.571832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42942 (local address 127.10.191.195:39807)
0620 23:15:41.571996 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:41.572001 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:41.572021 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:15:41.572187 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:41.572191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:41.572259 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:41.572362 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:41.573635 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:41.574363 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:41.575759 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:41.576157 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:41.576252 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:41.580418 (+ 4166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:41.580433 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:41.580449 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:41.580481 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:41.583352 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:41.583953 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:41.583960 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:41.583963 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:41.584097 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:41.584395 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:41.584399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:41.584400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:41.584732 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:41.584960 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:41.585170 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 23:15:41.585332 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 23:15:41.586194 28634 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:15:41.674331 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 28511
I20260620 23:15:41.690696 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 28151
I20260620 23:15:41.691908 28213 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:15:41.805636 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 28151
2026-06-20T23:15:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4254 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T23:15:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:15:41Z Disabled control of system clock
I20260620 23:15:41.884622 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:35647
--webserver_interface=127.10.191.254
--webserver_port=0
--builtin_ntp_servers=127.10.191.212:40461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:35647
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:15:42.208468 28692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:42.208736 28692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:42.208922 28692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:42.216125 28692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:15:42.216231 28692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:42.216315 28692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:42.216405 28692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:15:42.216485 28692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:15:42.225574 28692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:40461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:35647
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:35647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:15:42.226897 28692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:42.228662 28692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:42.239989 28697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:42.240062 28700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:42.240017 28698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:42.240829 28692 server_base.cc:1034] running on GCE node
I20260620 23:15:42.241585 28692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:42.242964 28692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:42.244199 28692 hybrid_clock.cc:648] HybridClock initialized: now 1781997342244139 us; error 49 us; skew 500 ppm
I20260620 23:15:42.244581 28692 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:15:42.250406 28692 webserver.cc:460] Webserver started at http://127.10.191.254:37037/ using document root <none> and password file <none>
I20260620 23:15:42.251173 28692 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:42.251302 28692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:42.251616 28692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:15:42.254464 28692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "c51c99cb4f1c4e288949a5806b8c7ce1"
format_stamp: "Formatted at 2026-06-20 23:15:42 on dist-test-slave-jkbq"
server_key: "1a954418f20b05993bf66cd5368a610c"
server_key_iv: "4f25c92abe7e3a3a4d5a6043fc2825e5"
server_key_version: "encryptionkey@0"
I20260620 23:15:42.255335 28692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "c51c99cb4f1c4e288949a5806b8c7ce1"
format_stamp: "Formatted at 2026-06-20 23:15:42 on dist-test-slave-jkbq"
server_key: "1a954418f20b05993bf66cd5368a610c"
server_key_iv: "4f25c92abe7e3a3a4d5a6043fc2825e5"
server_key_version: "encryptionkey@0"
I20260620 23:15:42.260416 28692 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 23:15:42.263947 28706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:42.265384 28692 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:15:42.265569 28692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "c51c99cb4f1c4e288949a5806b8c7ce1"
format_stamp: "Formatted at 2026-06-20 23:15:42 on dist-test-slave-jkbq"
server_key: "1a954418f20b05993bf66cd5368a610c"
server_key_iv: "4f25c92abe7e3a3a4d5a6043fc2825e5"
server_key_version: "encryptionkey@0"
I20260620 23:15:42.265789 28692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:42.289223 28692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:42.290264 28692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:42.290608 28692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:42.309158 28692 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:35647
I20260620 23:15:42.309178 28757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:35647 every 8 connection(s)
I20260620 23:15:42.310549 28692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:15:42.314419 28758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:15:42.316366 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 28692
I20260620 23:15:42.316546 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:15:42.317131 11007 external_mini_cluster.cc:1428] Setting key 30bf6e32d8212fb311dc46ff1ca04b26
I20260620 23:15:42.325986 28758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c51c99cb4f1c4e288949a5806b8c7ce1: Bootstrap starting.
I20260620 23:15:42.329042 28758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c51c99cb4f1c4e288949a5806b8c7ce1: Neither blocks nor log segments found. Creating new log.
I20260620 23:15:42.330178 28758 log.cc:824] T 00000000000000000000000000000000 P c51c99cb4f1c4e288949a5806b8c7ce1: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:42.330564 28761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:42.319158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58132 (local address 127.10.191.254:35647)
0620 23:15:42.320174 (+ 1016us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:42.320185 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:42.320227 (+ 42us) server_negotiation.cc:407] Connection header received
0620 23:15:42.321866 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:42.321893 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:42.322354 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:42.322719 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:42.323357 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:42.324664 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:42.326079 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:42.326794 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:42.327049 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:42.328276 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:42.328283 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:42.328296 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:15:42.328816 (+ 520us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:42.328906 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:15:42.328927 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:42.329057 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:42.329267 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 23:15:42.329575 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":395,"threads_started":1}
I20260620 23:15:42.333341 28758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c51c99cb4f1c4e288949a5806b8c7ce1: No bootstrap required, opened a new log
I20260620 23:15:42.340513 28758 raft_consensus.cc:348] T 00000000000000000000000000000000 P c51c99cb4f1c4e288949a5806b8c7ce1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51c99cb4f1c4e288949a5806b8c7ce1" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 35647 } }
I20260620 23:15:42.340832 28758 raft_consensus.cc:374] T 00000000000000000000000000000000 P c51c99cb4f1c4e288949a5806b8c7ce1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:15:42.340905 28758 raft_consensus.cc:729] T 00000000000000000000000000000000 P c51c99cb4f1c4e288949a5806b8c7ce1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c51c99cb4f1c4e288949a5806b8c7ce1, State: Initialized, Role: FOLLOWER
I20260620 23:15:42.341434 28758 consensus_queue.cc:260] T 00000000000000000000000000000000 P c51c99cb4f1c4e288949a5806b8c7ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51c99cb4f1c4e288949a5806b8c7ce1" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 35647 } }
I20260620 23:15:42.341672 28758 raft_consensus.cc:388] T 00000000000000000000000000000000 P c51c99cb4f1c4e288949a5806b8c7ce1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:15:42.341840 28758 raft_consensus.cc:482] T 00000000000000000000000000000000 P c51c99cb4f1c4e288949a5806b8c7ce1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:15:42.342027 28758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c51c99cb4f1c4e288949a5806b8c7ce1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:15:42.344216 28758 raft_consensus.cc:504] T 00000000000000000000000000000000 P c51c99cb4f1c4e288949a5806b8c7ce1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51c99cb4f1c4e288949a5806b8c7ce1" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 35647 } }
I20260620 23:15:42.344659 28758 leader_election.cc:302] T 00000000000000000000000000000000 P c51c99cb4f1c4e288949a5806b8c7ce1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c51c99cb4f1c4e288949a5806b8c7ce1; no voters:
I20260620 23:15:42.345217 28758 leader_election.cc:288] T 00000000000000000000000000000000 P c51c99cb4f1c4e288949a5806b8c7ce1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:15:42.345393 28763 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c51c99cb4f1c4e288949a5806b8c7ce1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:15:42.346063 28763 raft_consensus.cc:686] T 00000000000000000000000000000000 P c51c99cb4f1c4e288949a5806b8c7ce1 [term 1 LEADER]: Becoming Leader. State: Replica: c51c99cb4f1c4e288949a5806b8c7ce1, State: Running, Role: LEADER
I20260620 23:15:42.346560 28763 consensus_queue.cc:237] T 00000000000000000000000000000000 P c51c99cb4f1c4e288949a5806b8c7ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51c99cb4f1c4e288949a5806b8c7ce1" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 35647 } }
I20260620 23:15:42.347522 28758 sys_catalog.cc:564] T 00000000000000000000000000000000 P c51c99cb4f1c4e288949a5806b8c7ce1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:15:42.349635 28764 sys_catalog.cc:455] T 00000000000000000000000000000000 P c51c99cb4f1c4e288949a5806b8c7ce1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c51c99cb4f1c4e288949a5806b8c7ce1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51c99cb4f1c4e288949a5806b8c7ce1" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 35647 } } }
I20260620 23:15:42.349787 28765 sys_catalog.cc:455] T 00000000000000000000000000000000 P c51c99cb4f1c4e288949a5806b8c7ce1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c51c99cb4f1c4e288949a5806b8c7ce1. Latest consensus state: current_term: 1 leader_uuid: "c51c99cb4f1c4e288949a5806b8c7ce1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51c99cb4f1c4e288949a5806b8c7ce1" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 35647 } } }
I20260620 23:15:42.350129 28764 sys_catalog.cc:458] T 00000000000000000000000000000000 P c51c99cb4f1c4e288949a5806b8c7ce1 [sys.catalog]: This master's current role is: LEADER
I20260620 23:15:42.350122 28765 sys_catalog.cc:458] T 00000000000000000000000000000000 P c51c99cb4f1c4e288949a5806b8c7ce1 [sys.catalog]: This master's current role is: LEADER
I20260620 23:15:42.351853 28771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:15:42.358178 28771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:15:42.366113 28771 catalog_manager.cc:1347] Generated new cluster ID: 9117a5028e2643b2a44e1b67a9a9c02d
I20260620 23:15:42.366225 28771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:15:42.381291 28771 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:15:42.382977 28771 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:15:42.390370 28771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c51c99cb4f1c4e288949a5806b8c7ce1: Generated new TSK 0
I20260620 23:15:42.391391 28771 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:15:42.400986 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.193:0
--local_ip_for_outbound_sockets=127.10.191.193
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:35647
--builtin_ntp_servers=127.10.191.212:40461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:15:42.737015 28782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:42.737335 28782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:42.737500 28782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:42.744742 28782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:42.744876 28782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:42.745045 28782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.193
I20260620 23:15:42.754441 28782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:40461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:35647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.193
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:15:42.755817 28782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:42.757598 28782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:42.771052 28790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:42.771168 28788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:42.771268 28787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:42.772099 28782 server_base.cc:1034] running on GCE node
I20260620 23:15:42.772771 28782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:42.773779 28782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:42.774971 28782 hybrid_clock.cc:648] HybridClock initialized: now 1781997342774907 us; error 45 us; skew 500 ppm
I20260620 23:15:42.775327 28782 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:15:42.777576 28782 webserver.cc:460] Webserver started at http://127.10.191.193:34411/ using document root <none> and password file <none>
I20260620 23:15:42.778280 28782 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:42.778409 28782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:42.778733 28782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:15:42.781641 28782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-0/data/instance:
uuid: "53b5ff9a2aff49c58e7e94eaf8238760"
format_stamp: "Formatted at 2026-06-20 23:15:42 on dist-test-slave-jkbq"
server_key: "e4cc32da7c61dab1ca5a95714ad2cd2f"
server_key_iv: "6942cc171c0772f68e4195bdcb5fb870"
server_key_version: "encryptionkey@0"
I20260620 23:15:42.782452 28782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance:
uuid: "53b5ff9a2aff49c58e7e94eaf8238760"
format_stamp: "Formatted at 2026-06-20 23:15:42 on dist-test-slave-jkbq"
server_key: "e4cc32da7c61dab1ca5a95714ad2cd2f"
server_key_iv: "6942cc171c0772f68e4195bdcb5fb870"
server_key_version: "encryptionkey@0"
I20260620 23:15:42.787616 28782 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 23:15:42.791006 28796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:42.792418 28782 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:15:42.792601 28782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-0/wal
uuid: "53b5ff9a2aff49c58e7e94eaf8238760"
format_stamp: "Formatted at 2026-06-20 23:15:42 on dist-test-slave-jkbq"
server_key: "e4cc32da7c61dab1ca5a95714ad2cd2f"
server_key_iv: "6942cc171c0772f68e4195bdcb5fb870"
server_key_version: "encryptionkey@0"
I20260620 23:15:42.792819 28782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:42.818377 28782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:42.819288 28782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:42.819604 28782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:42.820611 28782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:15:42.822065 28782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:15:42.822230 28782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:42.822359 28782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:15:42.822451 28782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:42.855885 28782 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.193:33097
I20260620 23:15:42.855973 28908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.193:33097 every 8 connection(s)
I20260620 23:15:42.857201 28782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
I20260620 23:15:42.864760 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 28782
I20260620 23:15:42.864941 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance
I20260620 23:15:42.865494 11007 external_mini_cluster.cc:1428] Setting key cee618f0564bf09be070bf5b60f8e705
I20260620 23:15:42.868506 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.194:0
--local_ip_for_outbound_sockets=127.10.191.194
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:35647
--builtin_ntp_servers=127.10.191.212:40461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 23:15:42.870797 28911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:42.859779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:48861 (local address 127.10.191.254:35647)
0620 23:15:42.860169 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:42.860173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:42.861415 (+ 1242us) server_negotiation.cc:407] Connection header received
0620 23:15:42.863285 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:42.863290 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:42.863359 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:42.863465 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:42.864535 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:42.865626 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:42.867453 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:42.868049 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:42.868188 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:42.869712 (+ 1524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:42.869721 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:42.869729 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:15:42.869754 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:42.869810 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:15:42.869838 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:42.869931 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:42.870179 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 23:15:42.870285 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":197,"threads_started":1}
I20260620 23:15:42.871595 28912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:42.860393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:35647 (local address 127.10.191.193:48861)
0620 23:15:42.861250 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:42.861284 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:42.863010 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:42.863656 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:42.863663 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:42.863753 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:42.864322 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:42.864360 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:42.865772 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:42.865776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:42.867298 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:42.867312 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:42.868247 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:42.868252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:42.868517 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:42.869382 (+ 865us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:15:42.869386 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:42.869437 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:15:42.869534 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:42.869958 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:42.869969 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:42.869977 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 23:15:42.870133 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 23:15:42.870499 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":252,"threads_started":1}
I20260620 23:15:42.873093 28909 heartbeater.cc:347] Connected to a master server at 127.10.191.254:35647
I20260620 23:15:42.873413 28909 heartbeater.cc:464] Registering TS with master...
I20260620 23:15:42.874120 28909 heartbeater.cc:511] Master 127.10.191.254:35647 requested a full tablet report, sending...
I20260620 23:15:42.876137 28723 ts_manager.cc:194] Registered new tserver with Master: 53b5ff9a2aff49c58e7e94eaf8238760 (127.10.191.193:33097)
I20260620 23:15:42.877585 28723 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.191.193:48861
W20260620 23:15:43.189442 28913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:43.189721 28913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:43.189865 28913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:43.197111 28913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:43.197256 28913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:43.197454 28913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.194
I20260620 23:15:43.206403 28913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:40461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:35647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.194
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:15:43.207742 28913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:43.209482 28913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:43.222980 28919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:43.222855 28921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:43.223011 28918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:43.224171 28913 server_base.cc:1034] running on GCE node
I20260620 23:15:43.224786 28913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:43.225760 28913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:43.226976 28913 hybrid_clock.cc:648] HybridClock initialized: now 1781997343226937 us; error 49 us; skew 500 ppm
I20260620 23:15:43.227316 28913 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:15:43.229692 28913 webserver.cc:460] Webserver started at http://127.10.191.194:42431/ using document root <none> and password file <none>
I20260620 23:15:43.230394 28913 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:43.230530 28913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:43.230847 28913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:15:43.233700 28913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-1/data/instance:
uuid: "178b511cf62b44b1874e65fc92d9e1af"
format_stamp: "Formatted at 2026-06-20 23:15:43 on dist-test-slave-jkbq"
server_key: "ef216415fae11bbc5ad8c51eb2f63426"
server_key_iv: "18ac3a4b2600681a65122fe03937a43f"
server_key_version: "encryptionkey@0"
I20260620 23:15:43.234484 28913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance:
uuid: "178b511cf62b44b1874e65fc92d9e1af"
format_stamp: "Formatted at 2026-06-20 23:15:43 on dist-test-slave-jkbq"
server_key: "ef216415fae11bbc5ad8c51eb2f63426"
server_key_iv: "18ac3a4b2600681a65122fe03937a43f"
server_key_version: "encryptionkey@0"
I20260620 23:15:43.239630 28913 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260620 23:15:43.243057 28927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:43.244282 28913 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:15:43.244462 28913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-1/wal
uuid: "178b511cf62b44b1874e65fc92d9e1af"
format_stamp: "Formatted at 2026-06-20 23:15:43 on dist-test-slave-jkbq"
server_key: "ef216415fae11bbc5ad8c51eb2f63426"
server_key_iv: "18ac3a4b2600681a65122fe03937a43f"
server_key_version: "encryptionkey@0"
I20260620 23:15:43.244678 28913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:43.259698 28913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:43.260459 28913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:43.260763 28913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:43.261720 28913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:15:43.263103 28913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:15:43.263226 28913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:43.263361 28913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:15:43.263449 28913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:43.296279 28913 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.194:42041
I20260620 23:15:43.296396 29039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.194:42041 every 8 connection(s)
I20260620 23:15:43.297748 28913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
I20260620 23:15:43.302424 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 28913
I20260620 23:15:43.302623 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance
I20260620 23:15:43.303126 11007 external_mini_cluster.cc:1428] Setting key c50b4e3fd0cb319670f2ef3498dc1e0c
I20260620 23:15:43.305938 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.195:0
--local_ip_for_outbound_sockets=127.10.191.195
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:35647
--builtin_ntp_servers=127.10.191.212:40461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 23:15:43.310696 28911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:43.300443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:35949 (local address 127.10.191.254:35647)
0620 23:15:43.300621 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:43.300625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:43.301959 (+ 1334us) server_negotiation.cc:407] Connection header received
0620 23:15:43.304083 (+ 2124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:43.304089 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:43.304181 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:43.304304 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:43.305506 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:43.306313 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:43.307995 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:43.308424 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:43.308556 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:43.309895 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:43.309905 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:43.309908 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:15:43.309925 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:43.309956 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:15:43.309967 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:43.310059 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:43.310279 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 23:15:43.310396 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:15:43.311803 29042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:43.300937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:35647 (local address 127.10.191.194:35949)
0620 23:15:43.301802 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:43.301836 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:43.303786 (+ 1950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:43.304537 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:43.304546 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:43.304630 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:43.305343 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:43.305363 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:43.306471 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:43.306476 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:43.307848 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:43.307861 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:43.308654 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:43.308662 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:43.308885 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:43.309591 (+ 706us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:15:43.309594 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:43.309633 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:15:43.309705 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:43.310070 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:43.310082 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:43.310096 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 23:15:43.310278 (+ 182us) client_negotiation.cc:240] Negotiation successful
0620 23:15:43.310665 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":261,"threads_started":1}
I20260620 23:15:43.313182 29040 heartbeater.cc:347] Connected to a master server at 127.10.191.254:35647
I20260620 23:15:43.313488 29040 heartbeater.cc:464] Registering TS with master...
I20260620 23:15:43.314359 29040 heartbeater.cc:511] Master 127.10.191.254:35647 requested a full tablet report, sending...
I20260620 23:15:43.315918 28723 ts_manager.cc:194] Registered new tserver with Master: 178b511cf62b44b1874e65fc92d9e1af (127.10.191.194:42041)
I20260620 23:15:43.316817 28723 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.191.194:35949
W20260620 23:15:43.635711 29043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:43.635980 29043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:43.636148 29043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:43.643249 29043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:43.643340 29043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:43.643513 29043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.195
I20260620 23:15:43.652837 29043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:40461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:35647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.195
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:15:43.654098 29043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:43.655843 29043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:43.668462 29048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:43.668531 29051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:43.668504 29049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:43.669291 29043 server_base.cc:1034] running on GCE node
I20260620 23:15:43.669898 29043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:43.670832 29043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:43.672096 29043 hybrid_clock.cc:648] HybridClock initialized: now 1781997343672050 us; error 41 us; skew 500 ppm
I20260620 23:15:43.672401 29043 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:15:43.674557 29043 webserver.cc:460] Webserver started at http://127.10.191.195:45889/ using document root <none> and password file <none>
I20260620 23:15:43.675297 29043 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:43.675428 29043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:43.675742 29043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:15:43.678561 29043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-2/data/instance:
uuid: "efbbf9eec94947729e1b65b8bf9646c9"
format_stamp: "Formatted at 2026-06-20 23:15:43 on dist-test-slave-jkbq"
server_key: "bdc999847627c72ac1ee2ec0e1824212"
server_key_iv: "0fa3b110fa4685cacde09efe12e71ed6"
server_key_version: "encryptionkey@0"
I20260620 23:15:43.679373 29043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance:
uuid: "efbbf9eec94947729e1b65b8bf9646c9"
format_stamp: "Formatted at 2026-06-20 23:15:43 on dist-test-slave-jkbq"
server_key: "bdc999847627c72ac1ee2ec0e1824212"
server_key_iv: "0fa3b110fa4685cacde09efe12e71ed6"
server_key_version: "encryptionkey@0"
I20260620 23:15:43.684347 29043 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 23:15:43.687587 29057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:43.688832 29043 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 23:15:43.689008 29043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-2/wal
uuid: "efbbf9eec94947729e1b65b8bf9646c9"
format_stamp: "Formatted at 2026-06-20 23:15:43 on dist-test-slave-jkbq"
server_key: "bdc999847627c72ac1ee2ec0e1824212"
server_key_iv: "0fa3b110fa4685cacde09efe12e71ed6"
server_key_version: "encryptionkey@0"
I20260620 23:15:43.689231 29043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:43.706259 29043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:43.707160 29043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:43.707470 29043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:43.708417 29043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:15:43.709870 29043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:15:43.710000 29043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:43.710136 29043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:15:43.710232 29043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:43.743127 29043 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.195:46835
I20260620 23:15:43.743180 29169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.195:46835 every 8 connection(s)
I20260620 23:15:43.744455 29043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
I20260620 23:15:43.744917 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 29043
I20260620 23:15:43.745113 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance
I20260620 23:15:43.745738 11007 external_mini_cluster.cc:1428] Setting key 97e3b3ae5c0ded00ebc404eacba86838
I20260620 23:15:43.757002 28911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:43.747150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:52745 (local address 127.10.191.254:35647)
0620 23:15:43.747333 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:43.747338 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:43.749235 (+ 1897us) server_negotiation.cc:407] Connection header received
0620 23:15:43.750946 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:43.750951 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:43.751034 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:43.751173 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:43.752291 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:43.753092 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:43.754529 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:43.755003 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:43.755099 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:43.756372 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:43.756376 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:43.756379 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:15:43.756396 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:43.756423 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:15:43.756434 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:43.756499 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:43.756691 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 23:15:43.756763 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 23:15:43.757965 29172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:43.747935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:35647 (local address 127.10.191.195:52745)
0620 23:15:43.749080 (+ 1145us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:43.749127 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:43.750679 (+ 1552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:43.751398 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:43.751405 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:43.751491 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:43.752132 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:43.752150 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:43.753209 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:43.753213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:43.754404 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:43.754415 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:43.755185 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:43.755190 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:43.755387 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:43.756095 (+ 708us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:15:43.756098 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:43.756135 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:15:43.756214 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:43.756514 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:43.756521 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:43.756527 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 23:15:43.756661 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 23:15:43.756940 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":328,"threads_started":1}
I20260620 23:15:43.759475 29170 heartbeater.cc:347] Connected to a master server at 127.10.191.254:35647
I20260620 23:15:43.759780 29170 heartbeater.cc:464] Registering TS with master...
I20260620 23:15:43.760579 29170 heartbeater.cc:511] Master 127.10.191.254:35647 requested a full tablet report, sending...
I20260620 23:15:43.761938 28723 ts_manager.cc:194] Registered new tserver with Master: efbbf9eec94947729e1b65b8bf9646c9 (127.10.191.195:46835)
I20260620 23:15:43.762759 28723 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.191.195:52745
I20260620 23:15:43.773281 11007 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:15:43.783238 28911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:43.777378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58348 (local address 127.10.191.254:35647)
0620 23:15:43.777652 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:43.777658 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:43.777927 (+ 269us) server_negotiation.cc:407] Connection header received
0620 23:15:43.778132 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:43.778142 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:43.778218 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:43.778328 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:43.778817 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:43.779549 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:43.780880 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:43.781286 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:43.781420 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:43.782465 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:43.782470 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:43.782473 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:15:43.782489 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:43.782519 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:15:43.782531 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:43.782627 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:43.782772 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 23:15:43.782884 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 23:15:43.786844 28723 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58348:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:15:43.794047 28723 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:15:43.829298 29185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:43.807043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47774 (local address 127.10.191.195:46835)
0620 23:15:43.818229 (+ 11186us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:43.818242 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:43.819870 (+ 1628us) server_negotiation.cc:407] Connection header received
0620 23:15:43.820178 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:43.820185 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:43.820446 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:43.820614 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:43.822219 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:43.823927 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:43.827119 (+ 3192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:43.828166 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:43.828381 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:43.828605 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:43.828772 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 23:15:43.828940 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10899,"thread_start_us":10089,"threads_started":1}
I20260620 23:15:43.829341 29182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:43.806770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.195:46835 (local address 127.0.0.1:47774)
0620 23:15:43.819744 (+ 12974us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:43.819764 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:43.819975 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:43.820889 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:43.820893 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:43.820939 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:43.821553 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:43.821578 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:43.824108 (+ 2530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:43.824115 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:43.826916 (+ 2801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:43.826935 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:43.828355 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:43.828364 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:43.828538 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:43.828584 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 23:15:43.828717 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 23:15:43.828901 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12818,"mutex_wait_us":204,"spinlock_wait_cycles":7808,"thread_start_us":11688,"threads_started":1}
I20260620 23:15:43.830369 29184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:43.806598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.194:42041 (local address 127.0.0.1:33716)
0620 23:15:43.818888 (+ 12290us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:43.818931 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:43.819132 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:43.820884 (+ 1752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:43.820892 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:43.820932 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:43.822437 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:43.822453 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:43.824345 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:43.824355 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:43.827198 (+ 2843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:43.827209 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:43.829402 (+ 2193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:43.829410 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:43.829637 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:43.829700 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 23:15:43.829833 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 23:15:43.829960 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11997,"mutex_wait_us":844,"spinlock_wait_cycles":768,"thread_start_us":12840,"threads_started":1}
I20260620 23:15:43.830457 29187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:43.820180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52602 (local address 127.10.191.193:33097)
0620 23:15:43.821375 (+ 1195us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:43.821385 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:43.822599 (+ 1214us) server_negotiation.cc:407] Connection header received
0620 23:15:43.822712 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:43.822722 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:43.823002 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:43.823212 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:43.823728 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:43.824710 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:43.827484 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:43.828864 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:43.829025 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:43.829234 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:43.829934 (+ 700us) server_negotiation.cc:299] Negotiation successful
0620 23:15:43.830107 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1021,"thread_start_us":882,"threads_started":1}
I20260620 23:15:43.830463 29183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:43.806710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33716 (local address 127.10.191.194:42041)
0620 23:15:43.820041 (+ 13331us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:43.820050 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:43.820079 (+ 29us) server_negotiation.cc:407] Connection header received
0620 23:15:43.820171 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:43.820179 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:43.820448 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:43.820621 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:43.822669 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:43.824164 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:43.827378 (+ 3214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:43.828864 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:43.829028 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:43.829250 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:43.829912 (+ 662us) server_negotiation.cc:299] Negotiation successful
0620 23:15:43.830101 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13171,"thread_start_us":13064,"threads_started":1}
I20260620 23:15:43.831091 29186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:43.820164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:33097 (local address 127.0.0.1:52602)
0620 23:15:43.822347 (+ 2183us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:43.822364 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:43.822479 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:43.823199 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:43.823204 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:43.823230 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:43.823578 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:43.823586 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:43.824848 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:43.824860 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:43.827282 (+ 2422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:43.827297 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:43.829420 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:43.829426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:43.829638 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:43.829715 (+ 77us) client_negotiation.cc:769] Sending connection context
0620 23:15:43.829789 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 23:15:43.830745 (+ 956us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2056,"mutex_wait_us":1109,"thread_start_us":1752,"threads_started":1}
I20260620 23:15:43.834373 28844 tablet_service.cc:1467] Processing CreateTablet for tablet fb26220d3c6d46c5b97a1f5bacd52f25 (DEFAULT_TABLE table=test-table [id=f80c8d1b244c48b884af29c38d77aaef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:15:43.834529 28975 tablet_service.cc:1467] Processing CreateTablet for tablet fb26220d3c6d46c5b97a1f5bacd52f25 (DEFAULT_TABLE table=test-table [id=f80c8d1b244c48b884af29c38d77aaef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:15:43.834532 29105 tablet_service.cc:1467] Processing CreateTablet for tablet fb26220d3c6d46c5b97a1f5bacd52f25 (DEFAULT_TABLE table=test-table [id=f80c8d1b244c48b884af29c38d77aaef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:15:43.835781 28844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb26220d3c6d46c5b97a1f5bacd52f25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:15:43.835870 28975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb26220d3c6d46c5b97a1f5bacd52f25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:15:43.835868 29105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb26220d3c6d46c5b97a1f5bacd52f25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:15:43.847134 29188 tablet_bootstrap.cc:492] T fb26220d3c6d46c5b97a1f5bacd52f25 P 53b5ff9a2aff49c58e7e94eaf8238760: Bootstrap starting.
I20260620 23:15:43.851588 29188 tablet_bootstrap.cc:654] T fb26220d3c6d46c5b97a1f5bacd52f25 P 53b5ff9a2aff49c58e7e94eaf8238760: Neither blocks nor log segments found. Creating new log.
I20260620 23:15:43.852807 29188 log.cc:824] T fb26220d3c6d46c5b97a1f5bacd52f25 P 53b5ff9a2aff49c58e7e94eaf8238760: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:43.856070 29188 tablet_bootstrap.cc:492] T fb26220d3c6d46c5b97a1f5bacd52f25 P 53b5ff9a2aff49c58e7e94eaf8238760: No bootstrap required, opened a new log
I20260620 23:15:43.856374 29188 ts_tablet_manager.cc:1397] T fb26220d3c6d46c5b97a1f5bacd52f25 P 53b5ff9a2aff49c58e7e94eaf8238760: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.001s
I20260620 23:15:43.859017 29189 tablet_bootstrap.cc:492] T fb26220d3c6d46c5b97a1f5bacd52f25 P 178b511cf62b44b1874e65fc92d9e1af: Bootstrap starting.
I20260620 23:15:43.859329 29190 tablet_bootstrap.cc:492] T fb26220d3c6d46c5b97a1f5bacd52f25 P efbbf9eec94947729e1b65b8bf9646c9: Bootstrap starting.
I20260620 23:15:43.863025 29189 tablet_bootstrap.cc:654] T fb26220d3c6d46c5b97a1f5bacd52f25 P 178b511cf62b44b1874e65fc92d9e1af: Neither blocks nor log segments found. Creating new log.
I20260620 23:15:43.864239 29189 log.cc:824] T fb26220d3c6d46c5b97a1f5bacd52f25 P 178b511cf62b44b1874e65fc92d9e1af: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:43.866465 29188 raft_consensus.cc:348] T fb26220d3c6d46c5b97a1f5bacd52f25 P 53b5ff9a2aff49c58e7e94eaf8238760 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "178b511cf62b44b1874e65fc92d9e1af" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 42041 } } peers { permanent_uuid: "efbbf9eec94947729e1b65b8bf9646c9" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46835 } } peers { permanent_uuid: "53b5ff9a2aff49c58e7e94eaf8238760" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 33097 } }
I20260620 23:15:43.866994 29188 raft_consensus.cc:374] T fb26220d3c6d46c5b97a1f5bacd52f25 P 53b5ff9a2aff49c58e7e94eaf8238760 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:15:43.867139 29188 raft_consensus.cc:729] T fb26220d3c6d46c5b97a1f5bacd52f25 P 53b5ff9a2aff49c58e7e94eaf8238760 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53b5ff9a2aff49c58e7e94eaf8238760, State: Initialized, Role: FOLLOWER
I20260620 23:15:43.868384 29188 consensus_queue.cc:260] T fb26220d3c6d46c5b97a1f5bacd52f25 P 53b5ff9a2aff49c58e7e94eaf8238760 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "178b511cf62b44b1874e65fc92d9e1af" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 42041 } } peers { permanent_uuid: "efbbf9eec94947729e1b65b8bf9646c9" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46835 } } peers { permanent_uuid: "53b5ff9a2aff49c58e7e94eaf8238760" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 33097 } }
I20260620 23:15:43.869354 29190 tablet_bootstrap.cc:654] T fb26220d3c6d46c5b97a1f5bacd52f25 P efbbf9eec94947729e1b65b8bf9646c9: Neither blocks nor log segments found. Creating new log.
I20260620 23:15:43.871157 29190 log.cc:824] T fb26220d3c6d46c5b97a1f5bacd52f25 P efbbf9eec94947729e1b65b8bf9646c9: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:43.873333 29189 tablet_bootstrap.cc:492] T fb26220d3c6d46c5b97a1f5bacd52f25 P 178b511cf62b44b1874e65fc92d9e1af: No bootstrap required, opened a new log
I20260620 23:15:43.873621 29189 ts_tablet_manager.cc:1397] T fb26220d3c6d46c5b97a1f5bacd52f25 P 178b511cf62b44b1874e65fc92d9e1af: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.006s
I20260620 23:15:43.875674 28909 heartbeater.cc:503] Master 127.10.191.254:35647 was elected leader, sending a full tablet report...
I20260620 23:15:43.876899 29188 ts_tablet_manager.cc:1428] T fb26220d3c6d46c5b97a1f5bacd52f25 P 53b5ff9a2aff49c58e7e94eaf8238760: Time spent starting tablet: real 0.020s user 0.009s sys 0.010s
I20260620 23:15:43.883455 29189 raft_consensus.cc:348] T fb26220d3c6d46c5b97a1f5bacd52f25 P 178b511cf62b44b1874e65fc92d9e1af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "178b511cf62b44b1874e65fc92d9e1af" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 42041 } } peers { permanent_uuid: "efbbf9eec94947729e1b65b8bf9646c9" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46835 } } peers { permanent_uuid: "53b5ff9a2aff49c58e7e94eaf8238760" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 33097 } }
I20260620 23:15:43.883841 29189 raft_consensus.cc:374] T fb26220d3c6d46c5b97a1f5bacd52f25 P 178b511cf62b44b1874e65fc92d9e1af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:15:43.883953 29189 raft_consensus.cc:729] T fb26220d3c6d46c5b97a1f5bacd52f25 P 178b511cf62b44b1874e65fc92d9e1af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 178b511cf62b44b1874e65fc92d9e1af, State: Initialized, Role: FOLLOWER
I20260620 23:15:43.884343 29190 tablet_bootstrap.cc:492] T fb26220d3c6d46c5b97a1f5bacd52f25 P efbbf9eec94947729e1b65b8bf9646c9: No bootstrap required, opened a new log
I20260620 23:15:43.884619 29190 ts_tablet_manager.cc:1397] T fb26220d3c6d46c5b97a1f5bacd52f25 P efbbf9eec94947729e1b65b8bf9646c9: Time spent bootstrapping tablet: real 0.026s user 0.010s sys 0.007s
I20260620 23:15:43.884578 29189 consensus_queue.cc:260] T fb26220d3c6d46c5b97a1f5bacd52f25 P 178b511cf62b44b1874e65fc92d9e1af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "178b511cf62b44b1874e65fc92d9e1af" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 42041 } } peers { permanent_uuid: "efbbf9eec94947729e1b65b8bf9646c9" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46835 } } peers { permanent_uuid: "53b5ff9a2aff49c58e7e94eaf8238760" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 33097 } }
I20260620 23:15:43.885761 29040 heartbeater.cc:503] Master 127.10.191.254:35647 was elected leader, sending a full tablet report...
I20260620 23:15:43.886487 29189 ts_tablet_manager.cc:1428] T fb26220d3c6d46c5b97a1f5bacd52f25 P 178b511cf62b44b1874e65fc92d9e1af: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260620 23:15:43.892946 29190 raft_consensus.cc:348] T fb26220d3c6d46c5b97a1f5bacd52f25 P efbbf9eec94947729e1b65b8bf9646c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "178b511cf62b44b1874e65fc92d9e1af" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 42041 } } peers { permanent_uuid: "efbbf9eec94947729e1b65b8bf9646c9" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46835 } } peers { permanent_uuid: "53b5ff9a2aff49c58e7e94eaf8238760" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 33097 } }
I20260620 23:15:43.893309 29190 raft_consensus.cc:374] T fb26220d3c6d46c5b97a1f5bacd52f25 P efbbf9eec94947729e1b65b8bf9646c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:15:43.893415 29190 raft_consensus.cc:729] T fb26220d3c6d46c5b97a1f5bacd52f25 P efbbf9eec94947729e1b65b8bf9646c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efbbf9eec94947729e1b65b8bf9646c9, State: Initialized, Role: FOLLOWER
I20260620 23:15:43.893970 29190 consensus_queue.cc:260] T fb26220d3c6d46c5b97a1f5bacd52f25 P efbbf9eec94947729e1b65b8bf9646c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "178b511cf62b44b1874e65fc92d9e1af" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 42041 } } peers { permanent_uuid: "efbbf9eec94947729e1b65b8bf9646c9" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46835 } } peers { permanent_uuid: "53b5ff9a2aff49c58e7e94eaf8238760" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 33097 } }
I20260620 23:15:43.895015 29170 heartbeater.cc:503] Master 127.10.191.254:35647 was elected leader, sending a full tablet report...
I20260620 23:15:43.895650 29190 ts_tablet_manager.cc:1428] T fb26220d3c6d46c5b97a1f5bacd52f25 P efbbf9eec94947729e1b65b8bf9646c9: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260620 23:15:43.959141 29193 raft_consensus.cc:482] T fb26220d3c6d46c5b97a1f5bacd52f25 P 53b5ff9a2aff49c58e7e94eaf8238760 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:15:43.959450 29193 raft_consensus.cc:504] T fb26220d3c6d46c5b97a1f5bacd52f25 P 53b5ff9a2aff49c58e7e94eaf8238760 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "178b511cf62b44b1874e65fc92d9e1af" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 42041 } } peers { permanent_uuid: "efbbf9eec94947729e1b65b8bf9646c9" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46835 } } peers { permanent_uuid: "53b5ff9a2aff49c58e7e94eaf8238760" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 33097 } }
I20260620 23:15:43.961093 29193 leader_election.cc:288] T fb26220d3c6d46c5b97a1f5bacd52f25 P 53b5ff9a2aff49c58e7e94eaf8238760 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 178b511cf62b44b1874e65fc92d9e1af (127.10.191.194:42041), efbbf9eec94947729e1b65b8bf9646c9 (127.10.191.195:46835)
I20260620 23:15:43.968184 29185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:43.961633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:38701 (local address 127.10.191.195:46835)
0620 23:15:43.961847 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:43.961855 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:43.962842 (+ 987us) server_negotiation.cc:407] Connection header received
0620 23:15:43.963063 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:43.963068 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:43.963140 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:43.963236 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:43.963905 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:43.964616 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:43.966475 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:43.967302 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:43.967479 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:43.967549 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:43.967813 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 23:15:43.967949 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 23:15:43.968278 29198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:43.961560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.195:46835 (local address 127.10.191.193:38701)
0620 23:15:43.962719 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:43.962733 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:43.962912 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:43.963262 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:43.963266 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:43.963289 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:43.963727 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:43.963737 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:43.964754 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:43.964760 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:43.966275 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:43.966287 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:43.967486 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:43.967493 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:43.967655 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:43.967682 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 23:15:43.967787 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 23:15:43.967912 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1066,"mutex_wait_us":22,"thread_start_us":770,"threads_started":1}
I20260620 23:15:43.968372 29183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:43.961427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:39275 (local address 127.10.191.194:42041)
0620 23:15:43.961639 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:43.961647 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:43.962740 (+ 1093us) server_negotiation.cc:407] Connection header received
0620 23:15:43.963003 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:43.963009 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:43.963092 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:43.963192 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:43.963782 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:43.964484 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:43.966502 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:43.967510 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:43.967672 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:43.967738 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:43.968013 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 23:15:43.968136 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:15:43.968614 29197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:43.961414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.194:42041 (local address 127.10.191.193:39275)
0620 23:15:43.962620 (+ 1206us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:43.962638 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:43.962815 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:43.963211 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:43.963215 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:43.963245 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:43.963626 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:43.963636 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:43.964627 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:43.964635 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:43.966319 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:43.966333 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:43.967698 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:43.967705 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:43.967846 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:43.967868 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 23:15:43.967965 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 23:15:43.968085 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1076,"mutex_wait_us":98,"thread_start_us":690,"threads_started":1}
I20260620 23:15:43.969559 29125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb26220d3c6d46c5b97a1f5bacd52f25" candidate_uuid: "53b5ff9a2aff49c58e7e94eaf8238760" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efbbf9eec94947729e1b65b8bf9646c9" is_pre_election: true
I20260620 23:15:43.969604 28995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb26220d3c6d46c5b97a1f5bacd52f25" candidate_uuid: "53b5ff9a2aff49c58e7e94eaf8238760" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "178b511cf62b44b1874e65fc92d9e1af" is_pre_election: true
I20260620 23:15:43.970098 28995 raft_consensus.cc:2454] T fb26220d3c6d46c5b97a1f5bacd52f25 P 178b511cf62b44b1874e65fc92d9e1af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53b5ff9a2aff49c58e7e94eaf8238760 in term 0.
I20260620 23:15:43.970084 29125 raft_consensus.cc:2454] T fb26220d3c6d46c5b97a1f5bacd52f25 P efbbf9eec94947729e1b65b8bf9646c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53b5ff9a2aff49c58e7e94eaf8238760 in term 0.
I20260620 23:15:43.970743 28798 leader_election.cc:302] T fb26220d3c6d46c5b97a1f5bacd52f25 P 53b5ff9a2aff49c58e7e94eaf8238760 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53b5ff9a2aff49c58e7e94eaf8238760, efbbf9eec94947729e1b65b8bf9646c9; no voters:
I20260620 23:15:43.971458 29193 raft_consensus.cc:2781] T fb26220d3c6d46c5b97a1f5bacd52f25 P 53b5ff9a2aff49c58e7e94eaf8238760 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:15:43.971602 29193 raft_consensus.cc:482] T fb26220d3c6d46c5b97a1f5bacd52f25 P 53b5ff9a2aff49c58e7e94eaf8238760 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:15:43.971707 29193 raft_consensus.cc:3037] T fb26220d3c6d46c5b97a1f5bacd52f25 P 53b5ff9a2aff49c58e7e94eaf8238760 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:15:43.973845 29193 raft_consensus.cc:504] T fb26220d3c6d46c5b97a1f5bacd52f25 P 53b5ff9a2aff49c58e7e94eaf8238760 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "178b511cf62b44b1874e65fc92d9e1af" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 42041 } } peers { permanent_uuid: "efbbf9eec94947729e1b65b8bf9646c9" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46835 } } peers { permanent_uuid: "53b5ff9a2aff49c58e7e94eaf8238760" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 33097 } }
I20260620 23:15:43.974433 29193 leader_election.cc:288] T fb26220d3c6d46c5b97a1f5bacd52f25 P 53b5ff9a2aff49c58e7e94eaf8238760 [CANDIDATE]: Term 1 election: Requested vote from peers 178b511cf62b44b1874e65fc92d9e1af (127.10.191.194:42041), efbbf9eec94947729e1b65b8bf9646c9 (127.10.191.195:46835)
I20260620 23:15:43.974920 28995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb26220d3c6d46c5b97a1f5bacd52f25" candidate_uuid: "53b5ff9a2aff49c58e7e94eaf8238760" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "178b511cf62b44b1874e65fc92d9e1af"
I20260620 23:15:43.975158 28995 raft_consensus.cc:3037] T fb26220d3c6d46c5b97a1f5bacd52f25 P 178b511cf62b44b1874e65fc92d9e1af [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:15:43.975118 29125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb26220d3c6d46c5b97a1f5bacd52f25" candidate_uuid: "53b5ff9a2aff49c58e7e94eaf8238760" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efbbf9eec94947729e1b65b8bf9646c9"
I20260620 23:15:43.975349 29125 raft_consensus.cc:3037] T fb26220d3c6d46c5b97a1f5bacd52f25 P efbbf9eec94947729e1b65b8bf9646c9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:15:43.978452 28995 raft_consensus.cc:2454] T fb26220d3c6d46c5b97a1f5bacd52f25 P 178b511cf62b44b1874e65fc92d9e1af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53b5ff9a2aff49c58e7e94eaf8238760 in term 1.
I20260620 23:15:43.978498 29125 raft_consensus.cc:2454] T fb26220d3c6d46c5b97a1f5bacd52f25 P efbbf9eec94947729e1b65b8bf9646c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53b5ff9a2aff49c58e7e94eaf8238760 in term 1.
I20260620 23:15:43.978905 28799 leader_election.cc:302] T fb26220d3c6d46c5b97a1f5bacd52f25 P 53b5ff9a2aff49c58e7e94eaf8238760 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 178b511cf62b44b1874e65fc92d9e1af, 53b5ff9a2aff49c58e7e94eaf8238760; no voters:
I20260620 23:15:43.979218 29193 raft_consensus.cc:2781] T fb26220d3c6d46c5b97a1f5bacd52f25 P 53b5ff9a2aff49c58e7e94eaf8238760 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:15:43.979655 29193 raft_consensus.cc:686] T fb26220d3c6d46c5b97a1f5bacd52f25 P 53b5ff9a2aff49c58e7e94eaf8238760 [term 1 LEADER]: Becoming Leader. State: Replica: 53b5ff9a2aff49c58e7e94eaf8238760, State: Running, Role: LEADER
I20260620 23:15:43.980171 29193 consensus_queue.cc:237] T fb26220d3c6d46c5b97a1f5bacd52f25 P 53b5ff9a2aff49c58e7e94eaf8238760 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "178b511cf62b44b1874e65fc92d9e1af" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 42041 } } peers { permanent_uuid: "efbbf9eec94947729e1b65b8bf9646c9" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46835 } } peers { permanent_uuid: "53b5ff9a2aff49c58e7e94eaf8238760" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 33097 } }
I20260620 23:15:43.984416 28723 catalog_manager.cc:5495] T fb26220d3c6d46c5b97a1f5bacd52f25 P 53b5ff9a2aff49c58e7e94eaf8238760 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53b5ff9a2aff49c58e7e94eaf8238760 (127.10.191.193). New cstate: current_term: 1 leader_uuid: "53b5ff9a2aff49c58e7e94eaf8238760" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "178b511cf62b44b1874e65fc92d9e1af" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efbbf9eec94947729e1b65b8bf9646c9" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53b5ff9a2aff49c58e7e94eaf8238760" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 33097 } health_report { overall_health: HEALTHY } } }
W20260620 23:15:43.996187 29171 tablet.cc:2369] T fb26220d3c6d46c5b97a1f5bacd52f25 P efbbf9eec94947729e1b65b8bf9646c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:15:44.019764 29187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:44.014654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52606 (local address 127.10.191.193:33097)
0620 23:15:44.014834 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:44.014839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:44.014956 (+ 117us) server_negotiation.cc:407] Connection header received
0620 23:15:44.015144 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:44.015147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:44.015237 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:44.015376 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:15:44.015702 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:44.016399 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:44.017776 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:44.018174 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:44.018266 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:44.018737 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:15:44.019285 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:15:44.019408 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:44.019508 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 23:15:44.019568 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 23:15:44.027493 29125 raft_consensus.cc:1261] T fb26220d3c6d46c5b97a1f5bacd52f25 P efbbf9eec94947729e1b65b8bf9646c9 [term 1 FOLLOWER]: Refusing update from remote peer 53b5ff9a2aff49c58e7e94eaf8238760: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:15:44.027498 28995 raft_consensus.cc:1261] T fb26220d3c6d46c5b97a1f5bacd52f25 P 178b511cf62b44b1874e65fc92d9e1af [term 1 FOLLOWER]: Refusing update from remote peer 53b5ff9a2aff49c58e7e94eaf8238760: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:15:44.028421 29199 consensus_queue.cc:1035] T fb26220d3c6d46c5b97a1f5bacd52f25 P 53b5ff9a2aff49c58e7e94eaf8238760 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efbbf9eec94947729e1b65b8bf9646c9" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:15:44.028764 29193 consensus_queue.cc:1035] T fb26220d3c6d46c5b97a1f5bacd52f25 P 53b5ff9a2aff49c58e7e94eaf8238760 [LEADER]: Connected to new peer: Peer: permanent_uuid: "178b511cf62b44b1874e65fc92d9e1af" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:15:44.039871 29202 mvcc.cc:205] Tried to move back new op lower bound from 7299061121127231488 to 7299061120946327552. Current Snapshot: MvccSnapshot[applied={T|T < 7299061121127231488}]
I20260620 23:15:44.055073 28722 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:15:44.055379 28722 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:15:44.059167 28722 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c51c99cb4f1c4e288949a5806b8c7ce1: Sending DeleteTablet for 3 replicas of tablet fb26220d3c6d46c5b97a1f5bacd52f25
I20260620 23:15:44.060371 29105 tablet_service.cc:1514] Processing DeleteTablet for tablet fb26220d3c6d46c5b97a1f5bacd52f25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:15:44 UTC) from {username='slave'} at 127.0.0.1:47774
I20260620 23:15:44.060377 28844 tablet_service.cc:1514] Processing DeleteTablet for tablet fb26220d3c6d46c5b97a1f5bacd52f25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:15:44 UTC) from {username='slave'} at 127.0.0.1:52602
I20260620 23:15:44.060597 28975 tablet_service.cc:1514] Processing DeleteTablet for tablet fb26220d3c6d46c5b97a1f5bacd52f25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:15:44 UTC) from {username='slave'} at 127.0.0.1:33716
I20260620 23:15:44.062261 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 28782
I20260620 23:15:44.073001 29214 tablet_replica.cc:330] T fb26220d3c6d46c5b97a1f5bacd52f25 P 178b511cf62b44b1874e65fc92d9e1af: stopping tablet replica
I20260620 23:15:44.073608 29214 raft_consensus.cc:2229] T fb26220d3c6d46c5b97a1f5bacd52f25 P 178b511cf62b44b1874e65fc92d9e1af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:15:44.074249 29214 raft_consensus.cc:2258] T fb26220d3c6d46c5b97a1f5bacd52f25 P 178b511cf62b44b1874e65fc92d9e1af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:15:44.075425 29213 tablet_replica.cc:330] T fb26220d3c6d46c5b97a1f5bacd52f25 P 53b5ff9a2aff49c58e7e94eaf8238760: stopping tablet replica
I20260620 23:15:44.075871 29213 raft_consensus.cc:2229] T fb26220d3c6d46c5b97a1f5bacd52f25 P 53b5ff9a2aff49c58e7e94eaf8238760 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:15:44.076644 29213 raft_consensus.cc:2258] T fb26220d3c6d46c5b97a1f5bacd52f25 P 53b5ff9a2aff49c58e7e94eaf8238760 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:15:44.076901 29214 ts_tablet_manager.cc:1905] T fb26220d3c6d46c5b97a1f5bacd52f25 P 178b511cf62b44b1874e65fc92d9e1af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:15:44.078146 29212 tablet_replica.cc:330] T fb26220d3c6d46c5b97a1f5bacd52f25 P efbbf9eec94947729e1b65b8bf9646c9: stopping tablet replica
I20260620 23:15:44.078588 29212 raft_consensus.cc:2229] T fb26220d3c6d46c5b97a1f5bacd52f25 P efbbf9eec94947729e1b65b8bf9646c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:15:44.079134 29212 raft_consensus.cc:2258] T fb26220d3c6d46c5b97a1f5bacd52f25 P efbbf9eec94947729e1b65b8bf9646c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:15:44.079416 29213 ts_tablet_manager.cc:1905] T fb26220d3c6d46c5b97a1f5bacd52f25 P 53b5ff9a2aff49c58e7e94eaf8238760: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:15:44.083601 29212 ts_tablet_manager.cc:1905] T fb26220d3c6d46c5b97a1f5bacd52f25 P efbbf9eec94947729e1b65b8bf9646c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:15:44.086148 29187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:44.063894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52614 (local address 127.10.191.193:33097)
0620 23:15:44.064082 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:44.064088 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:44.077445 (+ 13357us) server_negotiation.cc:407] Connection header received
0620 23:15:44.077646 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:44.077652 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:44.077753 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:44.077884 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:44.078567 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:44.079679 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:44.082010 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:44.082698 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:44.082796 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:44.084342 (+ 1546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:44.084358 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:44.084374 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:15:44.085227 (+ 853us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:44.085283 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:15:44.085317 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:44.085405 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:44.085650 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 23:15:44.085764 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 23:15:44.088631 29214 ts_tablet_manager.cc:1918] T fb26220d3c6d46c5b97a1f5bacd52f25 P 178b511cf62b44b1874e65fc92d9e1af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:15:44.088832 29214 log.cc:1196] T fb26220d3c6d46c5b97a1f5bacd52f25 P 178b511cf62b44b1874e65fc92d9e1af: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-1/wal/wals/fb26220d3c6d46c5b97a1f5bacd52f25
I20260620 23:15:44.089282 29214 ts_tablet_manager.cc:1939] T fb26220d3c6d46c5b97a1f5bacd52f25 P 178b511cf62b44b1874e65fc92d9e1af: Deleting consensus metadata
I20260620 23:15:44.089476 29212 ts_tablet_manager.cc:1918] T fb26220d3c6d46c5b97a1f5bacd52f25 P efbbf9eec94947729e1b65b8bf9646c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:15:44.089620 29212 log.cc:1196] T fb26220d3c6d46c5b97a1f5bacd52f25 P efbbf9eec94947729e1b65b8bf9646c9: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-2/wal/wals/fb26220d3c6d46c5b97a1f5bacd52f25
I20260620 23:15:44.090049 29212 ts_tablet_manager.cc:1939] T fb26220d3c6d46c5b97a1f5bacd52f25 P efbbf9eec94947729e1b65b8bf9646c9: Deleting consensus metadata
I20260620 23:15:44.090842 29213 ts_tablet_manager.cc:1918] T fb26220d3c6d46c5b97a1f5bacd52f25 P 53b5ff9a2aff49c58e7e94eaf8238760: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:15:44.091084 28709 catalog_manager.cc:4841] TS 178b511cf62b44b1874e65fc92d9e1af (127.10.191.194:42041): tablet fb26220d3c6d46c5b97a1f5bacd52f25 (table test-table [id=f80c8d1b244c48b884af29c38d77aaef]) successfully deleted
I20260620 23:15:44.091092 29213 log.cc:1196] T fb26220d3c6d46c5b97a1f5bacd52f25 P 53b5ff9a2aff49c58e7e94eaf8238760: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781997042893763-11007-0/minicluster-data/ts-0/wal/wals/fb26220d3c6d46c5b97a1f5bacd52f25
I20260620 23:15:44.091518 28708 catalog_manager.cc:4841] TS efbbf9eec94947729e1b65b8bf9646c9 (127.10.191.195:46835): tablet fb26220d3c6d46c5b97a1f5bacd52f25 (table test-table [id=f80c8d1b244c48b884af29c38d77aaef]) successfully deleted
I20260620 23:15:44.091588 28904 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:15:44.185554 29213 ts_tablet_manager.cc:1939] T fb26220d3c6d46c5b97a1f5bacd52f25 P 53b5ff9a2aff49c58e7e94eaf8238760: Deleting consensus metadata
I20260620 23:15:44.186299 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 28782
W20260620 23:15:44.200690 28709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.191.193:33097 (error 108)
I20260620 23:15:44.201045 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 28913
I20260620 23:15:44.202754 29186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:44.202296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:33097 (local address 127.0.0.1:52628)
0620 23:15:44.202472 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:44.202600 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.193:33097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 23:15:44.202999 28709 catalog_manager.cc:4584] TS 53b5ff9a2aff49c58e7e94eaf8238760 (127.10.191.193:33097): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fb26220d3c6d46c5b97a1f5bacd52f25: Network error: Client connection negotiation failed: client connection to 127.10.191.193:33097: connect: Connection refused (error 111)
I20260620 23:15:44.207720 29183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:44.201876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33730 (local address 127.10.191.194:42041)
0620 23:15:44.202050 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:44.202054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:44.202086 (+ 32us) server_negotiation.cc:407] Connection header received
0620 23:15:44.202280 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:44.202286 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:44.202411 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:44.202517 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:44.203038 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:44.203862 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:44.205098 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:44.205502 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:44.205592 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:44.206487 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:44.206495 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:44.206508 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:15:44.207124 (+ 616us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:44.207174 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:15:44.207193 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:44.207278 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:44.207402 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 23:15:44.207484 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 23:15:44.208385 29035 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:15:44.256472 29186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:44.255994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:33097 (local address 127.0.0.1:52638)
0620 23:15:44.256215 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:44.256302 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.193:33097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 23:15:44.257563 29186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:44.257221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:33097 (local address 127.0.0.1:52654)
0620 23:15:44.257379 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:44.257444 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.193:33097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 23:15:44.299796 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 28913
I20260620 23:15:44.317369 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 29043
I20260620 23:15:44.318251 29186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:44.317712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:33097 (local address 127.0.0.1:52658)
0620 23:15:44.317949 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:44.318042 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.193:33097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 23:15:44.319772 29186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:44.319375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:33097 (local address 127.0.0.1:52668)
0620 23:15:44.319527 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:44.319626 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.193:33097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 23:15:44.325241 29185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:44.318312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47790 (local address 127.10.191.195:46835)
0620 23:15:44.318494 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:44.318500 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:44.318670 (+ 170us) server_negotiation.cc:407] Connection header received
0620 23:15:44.318955 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:44.318960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:44.319057 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:44.319142 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:44.319638 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:44.320448 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:44.322355 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:44.322714 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:44.322807 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:44.323916 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:44.323924 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:44.323939 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:15:44.324503 (+ 564us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:44.324550 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:15:44.324590 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:44.324674 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:44.324827 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 23:15:44.324909 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 23:15:44.325819 29165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:15:44.401108 29186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:44.400628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:33097 (local address 127.0.0.1:52672)
0620 23:15:44.400852 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:44.400957 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.193:33097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 23:15:44.402208 29186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:44.401794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:33097 (local address 127.0.0.1:52674)
0620 23:15:44.402021 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:44.402088 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.193:33097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260620 23:15:44.414261 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 29043
I20260620 23:15:44.431309 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 28692
I20260620 23:15:44.432368 28753 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:15:44.540640 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 28692
2026-06-20T23:15:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2729 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:15:44 dist-test-slave-jkbq krb5kdc[29223](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:15:44 dist-test-slave-jkbq krb5kdc[29223](info): set up 2 sockets
Jun 20 23:15:44 dist-test-slave-jkbq krb5kdc[29223](info): commencing operation
krb5kdc: starting...
W20260620 23:15:45.643015 11007 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Jun 20 23:15:45 dist-test-slave-jkbq krb5kdc[29223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:15:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:15:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.191.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
I20260620 23:15:45.878082 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:42205
--webserver_interface=127.10.191.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:36391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:42205
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:15:46.198594 29236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:46.198887 29236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:46.198980 29236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:46.206037 29236 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:15:46.206142 29236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:46.206233 29236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:46.206332 29236 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:15:46.206396 29236 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:15:46.216027 29236 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:36391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:42205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:42205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:15:46.217296 29236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:46.218647 29236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:46.230423 29241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:46.230443 29244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:46.230470 29242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:46.231442 29236 server_base.cc:1034] running on GCE node
I20260620 23:15:46.232125 29236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:46.233448 29236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:46.234660 29236 hybrid_clock.cc:648] HybridClock initialized: now 1781997346234615 us; error 35 us; skew 500 ppm
I20260620 23:15:46.235050 29236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:15:46 dist-test-slave-jkbq krb5kdc[29223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997346, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:46.238042 29236 init.cc:377] Logged in from keytab as kudu/127.10.191.254@KRBTEST.COM (short username kudu)
I20260620 23:15:46.239614 29236 webserver.cc:460] Webserver started at http://127.10.191.254:44179/ using document root <none> and password file <none>
I20260620 23:15:46.240281 29236 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:46.240411 29236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:46.240717 29236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:15:46.243685 29236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "1c7f1456831c4f3abadfc59befeb950c"
format_stamp: "Formatted at 2026-06-20 23:15:46 on dist-test-slave-jkbq"
server_key: "9ebdeaacc65de6d7bfb2d8d638c7acdc"
server_key_iv: "f3f8d79b509306d7f1ef1313e786bf34"
server_key_version: "encryptionkey@0"
I20260620 23:15:46.244426 29236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "1c7f1456831c4f3abadfc59befeb950c"
format_stamp: "Formatted at 2026-06-20 23:15:46 on dist-test-slave-jkbq"
server_key: "9ebdeaacc65de6d7bfb2d8d638c7acdc"
server_key_iv: "f3f8d79b509306d7f1ef1313e786bf34"
server_key_version: "encryptionkey@0"
I20260620 23:15:46.249495 29236 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 23:15:46.252789 29251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:46.254005 29236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:15:46.254187 29236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "1c7f1456831c4f3abadfc59befeb950c"
format_stamp: "Formatted at 2026-06-20 23:15:46 on dist-test-slave-jkbq"
server_key: "9ebdeaacc65de6d7bfb2d8d638c7acdc"
server_key_iv: "f3f8d79b509306d7f1ef1313e786bf34"
server_key_version: "encryptionkey@0"
I20260620 23:15:46.254372 29236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:46.268325 29236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:46.272341 29236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:46.272648 29236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:46.296337 29236 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:42205
I20260620 23:15:46.296373 29302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:42205 every 8 connection(s)
I20260620 23:15:46.297665 29236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:15:46.300630 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 29236
I20260620 23:15:46.300810 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:15:46.301378 11007 external_mini_cluster.cc:1428] Setting key b497c086ec77ccfd9598f2fc12ed86f6
I20260620 23:15:46.301631 29303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:15:46.315786 29303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c7f1456831c4f3abadfc59befeb950c: Bootstrap starting.
Jun 20 23:15:46 dist-test-slave-jkbq krb5kdc[29223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:15:46.319113 29303 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c7f1456831c4f3abadfc59befeb950c: Neither blocks nor log segments found. Creating new log.
I20260620 23:15:46.320257 29303 log.cc:824] T 00000000000000000000000000000000 P 1c7f1456831c4f3abadfc59befeb950c: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:46.322994 29303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c7f1456831c4f3abadfc59befeb950c: No bootstrap required, opened a new log
I20260620 23:15:46.325110 29306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:46.303222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43674 (local address 127.10.191.254:42205)
0620 23:15:46.303961 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:46.303972 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:46.304015 (+ 43us) server_negotiation.cc:407] Connection header received
0620 23:15:46.306323 (+ 2308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:46.306346 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:46.306771 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:46.307220 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:46.309000 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:46.310386 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:46.312313 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:46.313009 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:46.313186 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:46.317972 (+ 4786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:46.317979 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:46.317991 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:46.318030 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:46.321105 (+ 3075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:46.321797 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:46.321803 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:46.321809 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:46.321964 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:46.322368 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:46.322372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:46.322374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:46.322993 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:46.323286 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:46.323741 (+ 455us) server_negotiation.cc:299] Negotiation successful
0620 23:15:46.324152 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":348,"threads_started":1}
I20260620 23:15:46.331447 29303 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c7f1456831c4f3abadfc59befeb950c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7f1456831c4f3abadfc59befeb950c" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 42205 } }
I20260620 23:15:46.331820 29303 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c7f1456831c4f3abadfc59befeb950c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:15:46.331923 29303 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c7f1456831c4f3abadfc59befeb950c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c7f1456831c4f3abadfc59befeb950c, State: Initialized, Role: FOLLOWER
I20260620 23:15:46.332463 29303 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c7f1456831c4f3abadfc59befeb950c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7f1456831c4f3abadfc59befeb950c" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 42205 } }
I20260620 23:15:46.332695 29303 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c7f1456831c4f3abadfc59befeb950c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:15:46.332828 29303 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c7f1456831c4f3abadfc59befeb950c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:15:46.333009 29303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c7f1456831c4f3abadfc59befeb950c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:15:46.335160 29303 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c7f1456831c4f3abadfc59befeb950c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7f1456831c4f3abadfc59befeb950c" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 42205 } }
I20260620 23:15:46.335647 29303 leader_election.cc:302] T 00000000000000000000000000000000 P 1c7f1456831c4f3abadfc59befeb950c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c7f1456831c4f3abadfc59befeb950c; no voters:
I20260620 23:15:46.336256 29303 leader_election.cc:288] T 00000000000000000000000000000000 P 1c7f1456831c4f3abadfc59befeb950c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:15:46.336462 29308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c7f1456831c4f3abadfc59befeb950c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:15:46.337169 29308 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c7f1456831c4f3abadfc59befeb950c [term 1 LEADER]: Becoming Leader. State: Replica: 1c7f1456831c4f3abadfc59befeb950c, State: Running, Role: LEADER
I20260620 23:15:46.337705 29308 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c7f1456831c4f3abadfc59befeb950c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7f1456831c4f3abadfc59befeb950c" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 42205 } }
I20260620 23:15:46.338585 29303 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c7f1456831c4f3abadfc59befeb950c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:15:46.340675 29310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c7f1456831c4f3abadfc59befeb950c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c7f1456831c4f3abadfc59befeb950c. Latest consensus state: current_term: 1 leader_uuid: "1c7f1456831c4f3abadfc59befeb950c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7f1456831c4f3abadfc59befeb950c" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 42205 } } }
I20260620 23:15:46.341038 29310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c7f1456831c4f3abadfc59befeb950c [sys.catalog]: This master's current role is: LEADER
I20260620 23:15:46.342607 29315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:15:46.342568 29309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c7f1456831c4f3abadfc59befeb950c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c7f1456831c4f3abadfc59befeb950c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7f1456831c4f3abadfc59befeb950c" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 42205 } } }
I20260620 23:15:46.342893 29309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c7f1456831c4f3abadfc59befeb950c [sys.catalog]: This master's current role is: LEADER
I20260620 23:15:46.347780 29315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:15:46.357491 29315 catalog_manager.cc:1347] Generated new cluster ID: 25718eeea846443cbd67bc9952088458
I20260620 23:15:46.357599 29315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:15:46.405606 29315 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:15:46.407143 29315 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:15:46.417939 29315 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c7f1456831c4f3abadfc59befeb950c: Generated new TSK 0
I20260620 23:15:46.418814 29315 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.191.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab.
Entry for principal kudu/127.10.191.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab.
I20260620 23:15:46.498238 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.193:0
--local_ip_for_outbound_sockets=127.10.191.193
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:42205
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:36391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:15:46.836043 29329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:46.836323 29329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:46.836417 29329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:46.843966 29329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:46.844069 29329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:46.844189 29329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.193
I20260620 23:15:46.853942 29329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:36391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:42205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.193
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:15:46.855254 29329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:46.856554 29329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:46.870074 29334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:46.870167 29335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:46.870153 29337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:46.871249 29329 server_base.cc:1034] running on GCE node
I20260620 23:15:46.871882 29329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:46.872819 29329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:46.874076 29329 hybrid_clock.cc:648] HybridClock initialized: now 1781997346874048 us; error 37 us; skew 500 ppm
I20260620 23:15:46.874424 29329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:15:46 dist-test-slave-jkbq krb5kdc[29223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997346, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:46.877620 29329 init.cc:377] Logged in from keytab as kudu/127.10.191.193@KRBTEST.COM (short username kudu)
I20260620 23:15:46.879154 29329 webserver.cc:460] Webserver started at http://127.10.191.193:36271/ using document root <none> and password file <none>
I20260620 23:15:46.879802 29329 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:46.879901 29329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:46.880155 29329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:15:46.883123 29329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-0/data/instance:
uuid: "758f92f6c27041d6906bb109cf8e13c6"
format_stamp: "Formatted at 2026-06-20 23:15:46 on dist-test-slave-jkbq"
server_key: "20f76b19c1f66db032d45353eab56ccd"
server_key_iv: "043d6963b9f7397eca499241889e3941"
server_key_version: "encryptionkey@0"
I20260620 23:15:46.883901 29329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance:
uuid: "758f92f6c27041d6906bb109cf8e13c6"
format_stamp: "Formatted at 2026-06-20 23:15:46 on dist-test-slave-jkbq"
server_key: "20f76b19c1f66db032d45353eab56ccd"
server_key_iv: "043d6963b9f7397eca499241889e3941"
server_key_version: "encryptionkey@0"
I20260620 23:15:46.889094 29329 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260620 23:15:46.892721 29344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:46.894097 29329 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:15:46.894330 29329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-0/wal
uuid: "758f92f6c27041d6906bb109cf8e13c6"
format_stamp: "Formatted at 2026-06-20 23:15:46 on dist-test-slave-jkbq"
server_key: "20f76b19c1f66db032d45353eab56ccd"
server_key_iv: "043d6963b9f7397eca499241889e3941"
server_key_version: "encryptionkey@0"
I20260620 23:15:46.894508 29329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:46.912925 29329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:46.917224 29329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:46.917524 29329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:46.918496 29329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:15:46.919967 29329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:15:46.920077 29329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:46.920241 29329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:15:46.920322 29329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:46.953321 29329 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.193:35775
I20260620 23:15:46.953405 29456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.193:35775 every 8 connection(s)
I20260620 23:15:46.954630 29329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
I20260620 23:15:46.955915 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 29329
I20260620 23:15:46.956092 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance
I20260620 23:15:46.956627 11007 external_mini_cluster.cc:1428] Setting key 0add4133ebdc479a18fe7979c09f46e7
Jun 20 23:15:46 dist-test-slave-jkbq krb5kdc[29223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997346, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:15:46.975149 29459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:46.957521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:40407 (local address 127.10.191.254:42205)
0620 23:15:46.958044 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:46.958050 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:46.959262 (+ 1212us) server_negotiation.cc:407] Connection header received
0620 23:15:46.961141 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:46.961146 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:46.961248 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:46.961378 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:46.962994 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:46.963725 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:46.965330 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:46.965745 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:46.965879 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:46.969392 (+ 3513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:46.969397 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:46.969400 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:46.969433 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:46.972058 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:46.972810 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:46.972820 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:46.972823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:46.972943 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:46.973394 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:46.973399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:46.973401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:46.973660 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:46.973818 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:46.974525 (+ 707us) server_negotiation.cc:299] Negotiation successful
0620 23:15:46.974748 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":256,"threads_started":1}
I20260620 23:15:46.976140 29460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:46.958169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:42205 (local address 127.10.191.193:40407)
0620 23:15:46.959123 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:46.959181 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:46.960867 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:46.961621 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:46.961631 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:46.962191 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:46.962772 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:46.962791 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:46.963910 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:46.963914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:46.965161 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:46.965174 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:46.965999 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:46.966007 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:46.966232 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:46.966973 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:15:46.967019 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:46.969150 (+ 2131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:46.972213 (+ 3063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:46.972222 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:46.972239 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:46.972671 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:46.973055 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:46.973060 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:46.973064 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:46.973258 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:46.973788 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:46.973796 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:46.974152 (+ 356us) client_negotiation.cc:769] Sending connection context
0620 23:15:46.974422 (+ 270us) client_negotiation.cc:240] Negotiation successful
0620 23:15:46.974998 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":315,"threads_started":1}
I20260620 23:15:46.977747 29457 heartbeater.cc:347] Connected to a master server at 127.10.191.254:42205
I20260620 23:15:46.978060 29457 heartbeater.cc:464] Registering TS with master...
I20260620 23:15:46.978765 29457 heartbeater.cc:511] Master 127.10.191.254:42205 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.191.194@KRBTEST.COM; defaulting to no policy
I20260620 23:15:46.980993 29268 ts_manager.cc:194] Registered new tserver with Master: 758f92f6c27041d6906bb109cf8e13c6 (127.10.191.193:35775)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.194@KRBTEST.COM" created.
I20260620 23:15:46.982538 29268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.193@KRBTEST.COM'} at 127.10.191.193:40407
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab.
Entry for principal kudu/127.10.191.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab.
I20260620 23:15:47.009814 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.194:0
--local_ip_for_outbound_sockets=127.10.191.194
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:42205
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:36391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:15:47.325234 29463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:47.325477 29463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:47.325567 29463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:47.332484 29463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:47.332579 29463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:47.332684 29463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.194
I20260620 23:15:47.341779 29463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:36391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:42205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.194
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:15:47.342893 29463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:47.344092 29463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:47.356467 29469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:47.356549 29468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:47.356525 29471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:47.357507 29463 server_base.cc:1034] running on GCE node
I20260620 23:15:47.358062 29463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:47.359050 29463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:47.360265 29463 hybrid_clock.cc:648] HybridClock initialized: now 1781997347360206 us; error 49 us; skew 500 ppm
I20260620 23:15:47.360592 29463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:15:47 dist-test-slave-jkbq krb5kdc[29223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997347, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:47.369333 29463 init.cc:377] Logged in from keytab as kudu/127.10.191.194@KRBTEST.COM (short username kudu)
I20260620 23:15:47.370801 29463 webserver.cc:460] Webserver started at http://127.10.191.194:35213/ using document root <none> and password file <none>
I20260620 23:15:47.371536 29463 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:47.371668 29463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:47.371992 29463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:15:47.374818 29463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-1/data/instance:
uuid: "29d4991375dc4ecfa9f053a422317d76"
format_stamp: "Formatted at 2026-06-20 23:15:47 on dist-test-slave-jkbq"
server_key: "e96cdb04185d0b1c65f31e42f95cd011"
server_key_iv: "2a0240bdde17772b80db2a9d427dfa89"
server_key_version: "encryptionkey@0"
I20260620 23:15:47.375625 29463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance:
uuid: "29d4991375dc4ecfa9f053a422317d76"
format_stamp: "Formatted at 2026-06-20 23:15:47 on dist-test-slave-jkbq"
server_key: "e96cdb04185d0b1c65f31e42f95cd011"
server_key_iv: "2a0240bdde17772b80db2a9d427dfa89"
server_key_version: "encryptionkey@0"
I20260620 23:15:47.380545 29463 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 23:15:47.383818 29478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:47.385162 29463 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:15:47.385334 29463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-1/wal
uuid: "29d4991375dc4ecfa9f053a422317d76"
format_stamp: "Formatted at 2026-06-20 23:15:47 on dist-test-slave-jkbq"
server_key: "e96cdb04185d0b1c65f31e42f95cd011"
server_key_iv: "2a0240bdde17772b80db2a9d427dfa89"
server_key_version: "encryptionkey@0"
I20260620 23:15:47.385514 29463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:47.400672 29463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:47.405530 29463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:47.405810 29463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:47.406697 29463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:15:47.407980 29463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:15:47.408051 29463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:47.408187 29463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:15:47.408241 29463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:47.439241 29463 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.194:46257
I20260620 23:15:47.439338 29590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.194:46257 every 8 connection(s)
I20260620 23:15:47.440546 29463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
I20260620 23:15:47.441847 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 29463
I20260620 23:15:47.442031 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance
I20260620 23:15:47.442598 11007 external_mini_cluster.cc:1428] Setting key c346f12e327721364fd93468d376fa3b
Jun 20 23:15:47 dist-test-slave-jkbq krb5kdc[29223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997347, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:15:47.462450 29459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:47.443345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:42335 (local address 127.10.191.254:42205)
0620 23:15:47.444192 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:47.444196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:47.444922 (+ 726us) server_negotiation.cc:407] Connection header received
0620 23:15:47.446717 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:47.446723 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:47.446812 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:47.446949 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:47.448405 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:47.449114 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:47.450664 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:47.451201 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:47.451344 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:47.455318 (+ 3974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:47.455327 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:47.455335 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:47.455386 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:47.459127 (+ 3741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:47.459811 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:47.459819 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:47.459825 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:47.459988 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:47.460464 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:47.460472 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:47.460478 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:47.460856 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:47.461015 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:47.461687 (+ 672us) server_negotiation.cc:299] Negotiation successful
0620 23:15:47.461963 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":725}
I20260620 23:15:47.463136 29593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:47.443835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:42205 (local address 127.10.191.194:42335)
0620 23:15:47.444778 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:47.444822 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:47.446453 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:47.447173 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:47.447189 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:47.447682 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:47.448242 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:47.448259 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:47.449286 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:47.449290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:47.450512 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:47.450523 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:47.451426 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:47.451432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:47.451697 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:47.452390 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:15:47.452424 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:47.455095 (+ 2671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:47.459255 (+ 4160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:47.459262 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:47.459274 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:47.459691 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:47.460143 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:47.460149 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:47.460153 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:47.460346 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:47.461004 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:47.461010 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:47.461345 (+ 335us) client_negotiation.cc:769] Sending connection context
0620 23:15:47.461567 (+ 222us) client_negotiation.cc:240] Negotiation successful
0620 23:15:47.461959 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":299,"threads_started":1}
I20260620 23:15:47.464643 29591 heartbeater.cc:347] Connected to a master server at 127.10.191.254:42205
I20260620 23:15:47.464944 29591 heartbeater.cc:464] Registering TS with master...
I20260620 23:15:47.465646 29591 heartbeater.cc:511] Master 127.10.191.254:42205 requested a full tablet report, sending...
I20260620 23:15:47.467147 29268 ts_manager.cc:194] Registered new tserver with Master: 29d4991375dc4ecfa9f053a422317d76 (127.10.191.194:46257)
I20260620 23:15:47.468075 29268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.194@KRBTEST.COM'} at 127.10.191.194:42335
WARNING: no policy specified for kudu/127.10.191.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab.
Entry for principal kudu/127.10.191.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab.
I20260620 23:15:47.497869 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.195:0
--local_ip_for_outbound_sockets=127.10.191.195
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:42205
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:36391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:15:47.817297 29596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:47.817566 29596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:47.817746 29596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:47.824939 29596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:47.825039 29596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:47.825238 29596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.195
I20260620 23:15:47.834908 29596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:36391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:42205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.195
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:15:47.836162 29596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:47.837486 29596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:47.850512 29604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:47.850577 29601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:47.850615 29602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:47.851610 29596 server_base.cc:1034] running on GCE node
I20260620 23:15:47.852191 29596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:47.853209 29596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:47.854410 29596 hybrid_clock.cc:648] HybridClock initialized: now 1781997347854377 us; error 31 us; skew 500 ppm
I20260620 23:15:47.854715 29596 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:15:47 dist-test-slave-jkbq krb5kdc[29223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997347, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:47.857931 29596 init.cc:377] Logged in from keytab as kudu/127.10.191.195@KRBTEST.COM (short username kudu)
I20260620 23:15:47.859369 29596 webserver.cc:460] Webserver started at http://127.10.191.195:43351/ using document root <none> and password file <none>
I20260620 23:15:47.860030 29596 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:47.860152 29596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:47.860461 29596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:15:47.863319 29596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-2/data/instance:
uuid: "bf3eaa6d12ea42f7ac04a96bb3603db3"
format_stamp: "Formatted at 2026-06-20 23:15:47 on dist-test-slave-jkbq"
server_key: "ae384e3449f2efa950a0d9a72d6b6017"
server_key_iv: "8acfbbfa23fcc46def2a71e745a9545e"
server_key_version: "encryptionkey@0"
I20260620 23:15:47.864043 29596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance:
uuid: "bf3eaa6d12ea42f7ac04a96bb3603db3"
format_stamp: "Formatted at 2026-06-20 23:15:47 on dist-test-slave-jkbq"
server_key: "ae384e3449f2efa950a0d9a72d6b6017"
server_key_iv: "8acfbbfa23fcc46def2a71e745a9545e"
server_key_version: "encryptionkey@0"
I20260620 23:15:47.868906 29596 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 23:15:47.872151 29611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:47.873427 29596 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:15:47.873590 29596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-2/wal
uuid: "bf3eaa6d12ea42f7ac04a96bb3603db3"
format_stamp: "Formatted at 2026-06-20 23:15:47 on dist-test-slave-jkbq"
server_key: "ae384e3449f2efa950a0d9a72d6b6017"
server_key_iv: "8acfbbfa23fcc46def2a71e745a9545e"
server_key_version: "encryptionkey@0"
I20260620 23:15:47.873775 29596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:47.888713 29596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:47.892513 29596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:47.892787 29596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:47.893715 29596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:15:47.895041 29596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:15:47.895138 29596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:47.895277 29596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:15:47.895351 29596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:47.929301 29596 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.195:38499
I20260620 23:15:47.929392 29723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.195:38499 every 8 connection(s)
I20260620 23:15:47.930611 29596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
I20260620 23:15:47.939606 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 29596
I20260620 23:15:47.939812 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance
I20260620 23:15:47.940248 11007 external_mini_cluster.cc:1428] Setting key 8412641e63d8c5837a8af38d07414a3d
Jun 20 23:15:47 dist-test-slave-jkbq krb5kdc[29223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997347, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:15:47.949996 29459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:47.933095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:44375 (local address 127.10.191.254:42205)
0620 23:15:47.933276 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:47.933280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:47.934414 (+ 1134us) server_negotiation.cc:407] Connection header received
0620 23:15:47.936291 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:47.936297 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:47.936399 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:47.936510 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:47.937947 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:47.938758 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:47.940531 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:47.940944 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:47.941083 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:47.944475 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:47.944480 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:47.944483 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:47.944516 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:47.947170 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:47.947836 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:47.947844 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:47.947852 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:47.947964 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:47.948387 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:47.948391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:47.948393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:47.948649 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:47.948804 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:47.949456 (+ 652us) server_negotiation.cc:299] Negotiation successful
0620 23:15:47.949662 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:15:47.950927 29726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:47.933566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:42205 (local address 127.10.191.195:44375)
0620 23:15:47.934253 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:47.934291 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:47.936020 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:47.936733 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:47.936741 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:47.937235 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:47.937773 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:47.937790 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:47.938941 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:47.938945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:47.940398 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:47.940411 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:47.941161 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:47.941172 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:47.941423 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:47.942113 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:15:47.942145 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:47.944277 (+ 2132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:47.947301 (+ 3024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:47.947310 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:47.947325 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:47.947716 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:47.948069 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:47.948075 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:47.948079 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:47.948260 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:47.948772 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:47.948779 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:47.949091 (+ 312us) client_negotiation.cc:769] Sending connection context
0620 23:15:47.949341 (+ 250us) client_negotiation.cc:240] Negotiation successful
0620 23:15:47.949824 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":277,"threads_started":1}
I20260620 23:15:47.952199 29724 heartbeater.cc:347] Connected to a master server at 127.10.191.254:42205
I20260620 23:15:47.952478 29724 heartbeater.cc:464] Registering TS with master...
I20260620 23:15:47.953217 29724 heartbeater.cc:511] Master 127.10.191.254:42205 requested a full tablet report, sending...
I20260620 23:15:47.954797 29268 ts_manager.cc:194] Registered new tserver with Master: bf3eaa6d12ea42f7ac04a96bb3603db3 (127.10.191.195:38499)
I20260620 23:15:47.955680 29268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.195@KRBTEST.COM'} at 127.10.191.195:44375
I20260620 23:15:47.967181 11007 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:15:47.983283 29459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:47.971007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43686 (local address 127.10.191.254:42205)
0620 23:15:47.971217 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:47.971224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:47.971538 (+ 314us) server_negotiation.cc:407] Connection header received
0620 23:15:47.971667 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:47.971670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:47.971725 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:47.971802 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:47.973176 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:47.973871 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:47.975247 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:47.975623 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:47.975709 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:47.978237 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:47.978241 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:47.978245 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:47.978272 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:47.980787 (+ 2515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:47.981531 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:47.981535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:47.981538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:47.981655 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:47.982013 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:47.982016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:47.982018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:47.982289 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:47.982490 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:47.982758 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 23:15:47.982938 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 23:15:47.985533 29457 heartbeater.cc:503] Master 127.10.191.254:42205 was elected leader, sending a full tablet report...
I20260620 23:15:47.986971 29267 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43686:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:15:47.993976 29267 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:15:48.025969 29738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:48.005704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36142 (local address 127.10.191.195:38499)
0620 23:15:48.017496 (+ 11792us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:48.017509 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:48.017551 (+ 42us) server_negotiation.cc:407] Connection header received
0620 23:15:48.017675 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:48.017687 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:48.018023 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:48.018267 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:48.019130 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:48.020936 (+ 1806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:48.023321 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:48.024749 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:48.024941 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:48.025126 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:48.025447 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 23:15:48.025613 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11557,"thread_start_us":12793,"threads_started":1}
I20260620 23:15:48.026033 29736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:48.005397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.195:38499 (local address 127.0.0.1:36142)
0620 23:15:48.012385 (+ 6988us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:48.012446 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:48.012649 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:48.018279 (+ 5630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:48.018287 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:48.018321 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:48.018854 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:48.018891 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:48.021089 (+ 2198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:48.021096 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:48.023113 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:48.023130 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:48.024961 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:48.024968 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:48.025187 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:48.025235 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 23:15:48.025365 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 23:15:48.025546 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6711,"mutex_wait_us":3,"thread_start_us":6583,"threads_started":1}
I20260620 23:15:48.029024 29737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:48.005584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:35775 (local address 127.0.0.1:57770)
0620 23:15:48.019139 (+ 13555us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:48.019158 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:48.019278 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:48.021033 (+ 1755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:48.021038 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:48.021068 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:48.021620 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:48.021634 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:48.023652 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:48.023661 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:48.025928 (+ 2267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:48.025947 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:48.027884 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:48.027898 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:48.028054 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:48.028417 (+ 363us) client_negotiation.cc:769] Sending connection context
0620 23:15:48.028516 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 23:15:48.028649 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13427,"mutex_wait_us":430,"thread_start_us":13289,"threads_started":1}
I20260620 23:15:48.029507 29740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:48.013377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48942 (local address 127.10.191.194:46257)
0620 23:15:48.020583 (+ 7206us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:48.020591 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:48.020617 (+ 26us) server_negotiation.cc:407] Connection header received
0620 23:15:48.020717 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:48.020724 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:48.020942 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:48.021371 (+ 429us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:48.022023 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:48.023555 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:48.026229 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:48.028416 (+ 2187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:48.028573 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:48.028719 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:48.029016 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 23:15:48.029195 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7047,"thread_start_us":7113,"threads_started":1}
I20260620 23:15:48.029582 29741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:48.018535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.194:46257 (local address 127.0.0.1:48942)
0620 23:15:48.019571 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:48.019594 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:48.019733 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:48.021176 (+ 1443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:48.021184 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:48.021213 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:48.021832 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:48.021844 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:48.023697 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:48.023704 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:48.026025 (+ 2321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:48.026044 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:48.028633 (+ 2589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:48.028641 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:48.028817 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:48.028845 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 23:15:48.028944 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 23:15:48.029065 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":866,"mutex_wait_us":119,"thread_start_us":689,"threads_started":1}
I20260620 23:15:48.030366 29739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:48.006194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57770 (local address 127.10.191.193:35775)
0620 23:15:48.020453 (+ 14259us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:48.020463 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:48.020490 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:15:48.020586 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:48.020593 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:48.020821 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:48.021020 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:48.021804 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:48.023489 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:48.026138 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:48.027326 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:48.028197 (+ 871us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:48.028370 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:48.029890 (+ 1520us) server_negotiation.cc:299] Negotiation successful
0620 23:15:48.030046 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14086,"thread_start_us":14470,"threads_started":1}
I20260620 23:15:48.030701 29659 tablet_service.cc:1467] Processing CreateTablet for tablet 8555330edbe648af8040a82be14bb052 (DEFAULT_TABLE table=test-table [id=8405d9af4da4482d80c6130ea49dd7c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:15:48.032186 29659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8555330edbe648af8040a82be14bb052. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:15:48.032671 29526 tablet_service.cc:1467] Processing CreateTablet for tablet 8555330edbe648af8040a82be14bb052 (DEFAULT_TABLE table=test-table [id=8405d9af4da4482d80c6130ea49dd7c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:15:48.033862 29526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8555330edbe648af8040a82be14bb052. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:15:48.033767 29392 tablet_service.cc:1467] Processing CreateTablet for tablet 8555330edbe648af8040a82be14bb052 (DEFAULT_TABLE table=test-table [id=8405d9af4da4482d80c6130ea49dd7c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:15:48.035068 29392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8555330edbe648af8040a82be14bb052. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:15:48.044497 29742 tablet_bootstrap.cc:492] T 8555330edbe648af8040a82be14bb052 P 29d4991375dc4ecfa9f053a422317d76: Bootstrap starting.
I20260620 23:15:48.048802 29742 tablet_bootstrap.cc:654] T 8555330edbe648af8040a82be14bb052 P 29d4991375dc4ecfa9f053a422317d76: Neither blocks nor log segments found. Creating new log.
I20260620 23:15:48.049647 29743 tablet_bootstrap.cc:492] T 8555330edbe648af8040a82be14bb052 P bf3eaa6d12ea42f7ac04a96bb3603db3: Bootstrap starting.
I20260620 23:15:48.050112 29742 log.cc:824] T 8555330edbe648af8040a82be14bb052 P 29d4991375dc4ecfa9f053a422317d76: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:48.051685 29744 tablet_bootstrap.cc:492] T 8555330edbe648af8040a82be14bb052 P 758f92f6c27041d6906bb109cf8e13c6: Bootstrap starting.
I20260620 23:15:48.053218 29742 tablet_bootstrap.cc:492] T 8555330edbe648af8040a82be14bb052 P 29d4991375dc4ecfa9f053a422317d76: No bootstrap required, opened a new log
I20260620 23:15:48.053484 29742 ts_tablet_manager.cc:1397] T 8555330edbe648af8040a82be14bb052 P 29d4991375dc4ecfa9f053a422317d76: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 23:15:48.053580 29743 tablet_bootstrap.cc:654] T 8555330edbe648af8040a82be14bb052 P bf3eaa6d12ea42f7ac04a96bb3603db3: Neither blocks nor log segments found. Creating new log.
I20260620 23:15:48.054786 29743 log.cc:824] T 8555330edbe648af8040a82be14bb052 P bf3eaa6d12ea42f7ac04a96bb3603db3: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:48.055939 29744 tablet_bootstrap.cc:654] T 8555330edbe648af8040a82be14bb052 P 758f92f6c27041d6906bb109cf8e13c6: Neither blocks nor log segments found. Creating new log.
I20260620 23:15:48.057121 29744 log.cc:824] T 8555330edbe648af8040a82be14bb052 P 758f92f6c27041d6906bb109cf8e13c6: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:48.057855 29743 tablet_bootstrap.cc:492] T 8555330edbe648af8040a82be14bb052 P bf3eaa6d12ea42f7ac04a96bb3603db3: No bootstrap required, opened a new log
I20260620 23:15:48.058168 29743 ts_tablet_manager.cc:1397] T 8555330edbe648af8040a82be14bb052 P bf3eaa6d12ea42f7ac04a96bb3603db3: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 23:15:48.061301 29744 tablet_bootstrap.cc:492] T 8555330edbe648af8040a82be14bb052 P 758f92f6c27041d6906bb109cf8e13c6: No bootstrap required, opened a new log
I20260620 23:15:48.061546 29744 ts_tablet_manager.cc:1397] T 8555330edbe648af8040a82be14bb052 P 758f92f6c27041d6906bb109cf8e13c6: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260620 23:15:48.061883 29742 raft_consensus.cc:348] T 8555330edbe648af8040a82be14bb052 P 29d4991375dc4ecfa9f053a422317d76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3eaa6d12ea42f7ac04a96bb3603db3" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 38499 } } peers { permanent_uuid: "758f92f6c27041d6906bb109cf8e13c6" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 35775 } } peers { permanent_uuid: "29d4991375dc4ecfa9f053a422317d76" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 46257 } }
I20260620 23:15:48.062256 29742 raft_consensus.cc:374] T 8555330edbe648af8040a82be14bb052 P 29d4991375dc4ecfa9f053a422317d76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:15:48.062354 29742 raft_consensus.cc:729] T 8555330edbe648af8040a82be14bb052 P 29d4991375dc4ecfa9f053a422317d76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d4991375dc4ecfa9f053a422317d76, State: Initialized, Role: FOLLOWER
I20260620 23:15:48.062969 29742 consensus_queue.cc:260] T 8555330edbe648af8040a82be14bb052 P 29d4991375dc4ecfa9f053a422317d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3eaa6d12ea42f7ac04a96bb3603db3" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 38499 } } peers { permanent_uuid: "758f92f6c27041d6906bb109cf8e13c6" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 35775 } } peers { permanent_uuid: "29d4991375dc4ecfa9f053a422317d76" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 46257 } }
I20260620 23:15:48.064055 29591 heartbeater.cc:503] Master 127.10.191.254:42205 was elected leader, sending a full tablet report...
I20260620 23:15:48.064527 29742 ts_tablet_manager.cc:1428] T 8555330edbe648af8040a82be14bb052 P 29d4991375dc4ecfa9f053a422317d76: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260620 23:15:48.069377 29743 raft_consensus.cc:348] T 8555330edbe648af8040a82be14bb052 P bf3eaa6d12ea42f7ac04a96bb3603db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3eaa6d12ea42f7ac04a96bb3603db3" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 38499 } } peers { permanent_uuid: "758f92f6c27041d6906bb109cf8e13c6" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 35775 } } peers { permanent_uuid: "29d4991375dc4ecfa9f053a422317d76" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 46257 } }
I20260620 23:15:48.069767 29743 raft_consensus.cc:374] T 8555330edbe648af8040a82be14bb052 P bf3eaa6d12ea42f7ac04a96bb3603db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:15:48.069904 29743 raft_consensus.cc:729] T 8555330edbe648af8040a82be14bb052 P bf3eaa6d12ea42f7ac04a96bb3603db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf3eaa6d12ea42f7ac04a96bb3603db3, State: Initialized, Role: FOLLOWER
I20260620 23:15:48.070497 29743 consensus_queue.cc:260] T 8555330edbe648af8040a82be14bb052 P bf3eaa6d12ea42f7ac04a96bb3603db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3eaa6d12ea42f7ac04a96bb3603db3" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 38499 } } peers { permanent_uuid: "758f92f6c27041d6906bb109cf8e13c6" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 35775 } } peers { permanent_uuid: "29d4991375dc4ecfa9f053a422317d76" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 46257 } }
I20260620 23:15:48.071593 29724 heartbeater.cc:503] Master 127.10.191.254:42205 was elected leader, sending a full tablet report...
I20260620 23:15:48.071539 29744 raft_consensus.cc:348] T 8555330edbe648af8040a82be14bb052 P 758f92f6c27041d6906bb109cf8e13c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3eaa6d12ea42f7ac04a96bb3603db3" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 38499 } } peers { permanent_uuid: "758f92f6c27041d6906bb109cf8e13c6" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 35775 } } peers { permanent_uuid: "29d4991375dc4ecfa9f053a422317d76" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 46257 } }
I20260620 23:15:48.071923 29744 raft_consensus.cc:374] T 8555330edbe648af8040a82be14bb052 P 758f92f6c27041d6906bb109cf8e13c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:15:48.072034 29744 raft_consensus.cc:729] T 8555330edbe648af8040a82be14bb052 P 758f92f6c27041d6906bb109cf8e13c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 758f92f6c27041d6906bb109cf8e13c6, State: Initialized, Role: FOLLOWER
I20260620 23:15:48.072136 29743 ts_tablet_manager.cc:1428] T 8555330edbe648af8040a82be14bb052 P bf3eaa6d12ea42f7ac04a96bb3603db3: Time spent starting tablet: real 0.014s user 0.012s sys 0.002s
I20260620 23:15:48.072582 29744 consensus_queue.cc:260] T 8555330edbe648af8040a82be14bb052 P 758f92f6c27041d6906bb109cf8e13c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3eaa6d12ea42f7ac04a96bb3603db3" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 38499 } } peers { permanent_uuid: "758f92f6c27041d6906bb109cf8e13c6" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 35775 } } peers { permanent_uuid: "29d4991375dc4ecfa9f053a422317d76" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 46257 } }
I20260620 23:15:48.074203 29744 ts_tablet_manager.cc:1428] T 8555330edbe648af8040a82be14bb052 P 758f92f6c27041d6906bb109cf8e13c6: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
W20260620 23:15:48.182252 29725 tablet.cc:2369] T 8555330edbe648af8040a82be14bb052 P bf3eaa6d12ea42f7ac04a96bb3603db3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:15:48.192701 29592 tablet.cc:2369] T 8555330edbe648af8040a82be14bb052 P 29d4991375dc4ecfa9f053a422317d76: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:15:48.207052 29458 tablet.cc:2369] T 8555330edbe648af8040a82be14bb052 P 758f92f6c27041d6906bb109cf8e13c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:15:48.243666 29748 raft_consensus.cc:482] T 8555330edbe648af8040a82be14bb052 P 29d4991375dc4ecfa9f053a422317d76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:15:48.243953 29748 raft_consensus.cc:504] T 8555330edbe648af8040a82be14bb052 P 29d4991375dc4ecfa9f053a422317d76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3eaa6d12ea42f7ac04a96bb3603db3" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 38499 } } peers { permanent_uuid: "758f92f6c27041d6906bb109cf8e13c6" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 35775 } } peers { permanent_uuid: "29d4991375dc4ecfa9f053a422317d76" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 46257 } }
I20260620 23:15:48.245543 29748 leader_election.cc:288] T 8555330edbe648af8040a82be14bb052 P 29d4991375dc4ecfa9f053a422317d76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf3eaa6d12ea42f7ac04a96bb3603db3 (127.10.191.195:38499), 758f92f6c27041d6906bb109cf8e13c6 (127.10.191.193:35775)
I20260620 23:15:48.253988 29738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:48.245971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:49417 (local address 127.10.191.195:38499)
0620 23:15:48.246248 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:48.246257 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:48.246709 (+ 452us) server_negotiation.cc:407] Connection header received
0620 23:15:48.246947 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:48.246952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:48.247023 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:48.247110 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:48.247863 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:48.248972 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:48.252059 (+ 3087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:48.253040 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:48.253159 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:48.253218 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:48.253604 (+ 386us) server_negotiation.cc:299] Negotiation successful
0620 23:15:48.253731 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260620 23:15:48.254107 29739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:48.245993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:56331 (local address 127.10.191.193:35775)
0620 23:15:48.246201 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:48.246209 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:48.246799 (+ 590us) server_negotiation.cc:407] Connection header received
0620 23:15:48.246923 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:48.246928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:48.247009 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:48.247110 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:48.247864 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:48.248957 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:48.252046 (+ 3089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:48.253024 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:48.253164 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:48.253250 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:48.253724 (+ 474us) server_negotiation.cc:299] Negotiation successful
0620 23:15:48.253851 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 23:15:48.254225 29752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:48.245967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:35775 (local address 127.10.191.194:56331)
0620 23:15:48.246683 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:48.246697 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:48.246786 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:48.247144 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:48.247148 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:48.247199 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:48.247715 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:48.247723 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:48.249102 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:48.249107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:48.251844 (+ 2737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:48.251861 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:48.253392 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:48.253396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:48.253568 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:48.253588 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 23:15:48.253693 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 23:15:48.253817 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"mutex_wait_us":154,"spinlock_wait_cycles":25344,"thread_start_us":414,"threads_started":1}
I20260620 23:15:48.254528 29751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:48.245850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.195:38499 (local address 127.10.191.194:49417)
0620 23:15:48.246584 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:48.246604 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:48.246772 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:48.247143 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:48.247148 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:48.247175 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:48.247710 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:48.247723 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:48.249101 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:48.249107 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:48.251845 (+ 2738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:48.251861 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:48.253239 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:48.253245 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:48.253409 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:48.253439 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 23:15:48.253575 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 23:15:48.253710 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"mutex_wait_us":144,"thread_start_us":408,"threads_started":1}
I20260620 23:15:48.255527 29412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8555330edbe648af8040a82be14bb052" candidate_uuid: "29d4991375dc4ecfa9f053a422317d76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "758f92f6c27041d6906bb109cf8e13c6" is_pre_election: true
I20260620 23:15:48.255620 29679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8555330edbe648af8040a82be14bb052" candidate_uuid: "29d4991375dc4ecfa9f053a422317d76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf3eaa6d12ea42f7ac04a96bb3603db3" is_pre_election: true
I20260620 23:15:48.256064 29412 raft_consensus.cc:2454] T 8555330edbe648af8040a82be14bb052 P 758f92f6c27041d6906bb109cf8e13c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d4991375dc4ecfa9f053a422317d76 in term 0.
I20260620 23:15:48.256145 29679 raft_consensus.cc:2454] T 8555330edbe648af8040a82be14bb052 P bf3eaa6d12ea42f7ac04a96bb3603db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d4991375dc4ecfa9f053a422317d76 in term 0.
I20260620 23:15:48.256793 29480 leader_election.cc:302] T 8555330edbe648af8040a82be14bb052 P 29d4991375dc4ecfa9f053a422317d76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 29d4991375dc4ecfa9f053a422317d76, 758f92f6c27041d6906bb109cf8e13c6, bf3eaa6d12ea42f7ac04a96bb3603db3; no voters:
I20260620 23:15:48.257244 29748 raft_consensus.cc:2781] T 8555330edbe648af8040a82be14bb052 P 29d4991375dc4ecfa9f053a422317d76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:15:48.257388 29748 raft_consensus.cc:482] T 8555330edbe648af8040a82be14bb052 P 29d4991375dc4ecfa9f053a422317d76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:15:48.257504 29748 raft_consensus.cc:3037] T 8555330edbe648af8040a82be14bb052 P 29d4991375dc4ecfa9f053a422317d76 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:15:48.259806 29748 raft_consensus.cc:504] T 8555330edbe648af8040a82be14bb052 P 29d4991375dc4ecfa9f053a422317d76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3eaa6d12ea42f7ac04a96bb3603db3" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 38499 } } peers { permanent_uuid: "758f92f6c27041d6906bb109cf8e13c6" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 35775 } } peers { permanent_uuid: "29d4991375dc4ecfa9f053a422317d76" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 46257 } }
I20260620 23:15:48.260474 29748 leader_election.cc:288] T 8555330edbe648af8040a82be14bb052 P 29d4991375dc4ecfa9f053a422317d76 [CANDIDATE]: Term 1 election: Requested vote from peers bf3eaa6d12ea42f7ac04a96bb3603db3 (127.10.191.195:38499), 758f92f6c27041d6906bb109cf8e13c6 (127.10.191.193:35775)
I20260620 23:15:48.260830 29679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8555330edbe648af8040a82be14bb052" candidate_uuid: "29d4991375dc4ecfa9f053a422317d76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf3eaa6d12ea42f7ac04a96bb3603db3"
I20260620 23:15:48.260970 29412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8555330edbe648af8040a82be14bb052" candidate_uuid: "29d4991375dc4ecfa9f053a422317d76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "758f92f6c27041d6906bb109cf8e13c6"
I20260620 23:15:48.261032 29679 raft_consensus.cc:3037] T 8555330edbe648af8040a82be14bb052 P bf3eaa6d12ea42f7ac04a96bb3603db3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:15:48.261207 29412 raft_consensus.cc:3037] T 8555330edbe648af8040a82be14bb052 P 758f92f6c27041d6906bb109cf8e13c6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:15:48.263353 29679 raft_consensus.cc:2454] T 8555330edbe648af8040a82be14bb052 P bf3eaa6d12ea42f7ac04a96bb3603db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d4991375dc4ecfa9f053a422317d76 in term 1.
I20260620 23:15:48.263510 29412 raft_consensus.cc:2454] T 8555330edbe648af8040a82be14bb052 P 758f92f6c27041d6906bb109cf8e13c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d4991375dc4ecfa9f053a422317d76 in term 1.
I20260620 23:15:48.263762 29482 leader_election.cc:302] T 8555330edbe648af8040a82be14bb052 P 29d4991375dc4ecfa9f053a422317d76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d4991375dc4ecfa9f053a422317d76, bf3eaa6d12ea42f7ac04a96bb3603db3; no voters:
I20260620 23:15:48.264050 29748 raft_consensus.cc:2781] T 8555330edbe648af8040a82be14bb052 P 29d4991375dc4ecfa9f053a422317d76 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:15:48.264472 29748 raft_consensus.cc:686] T 8555330edbe648af8040a82be14bb052 P 29d4991375dc4ecfa9f053a422317d76 [term 1 LEADER]: Becoming Leader. State: Replica: 29d4991375dc4ecfa9f053a422317d76, State: Running, Role: LEADER
I20260620 23:15:48.264954 29748 consensus_queue.cc:237] T 8555330edbe648af8040a82be14bb052 P 29d4991375dc4ecfa9f053a422317d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3eaa6d12ea42f7ac04a96bb3603db3" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 38499 } } peers { permanent_uuid: "758f92f6c27041d6906bb109cf8e13c6" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 35775 } } peers { permanent_uuid: "29d4991375dc4ecfa9f053a422317d76" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 46257 } }
I20260620 23:15:48.269639 29267 catalog_manager.cc:5495] T 8555330edbe648af8040a82be14bb052 P 29d4991375dc4ecfa9f053a422317d76 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29d4991375dc4ecfa9f053a422317d76 (127.10.191.194). New cstate: current_term: 1 leader_uuid: "29d4991375dc4ecfa9f053a422317d76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3eaa6d12ea42f7ac04a96bb3603db3" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 38499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "758f92f6c27041d6906bb109cf8e13c6" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 35775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29d4991375dc4ecfa9f053a422317d76" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 46257 } health_report { overall_health: HEALTHY } } }
I20260620 23:15:48.321178 29740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:48.315819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48954 (local address 127.10.191.194:46257)
0620 23:15:48.316048 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:48.316056 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:48.316076 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:15:48.316175 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:48.316179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:48.316260 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:48.316422 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:15:48.316806 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:48.317541 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:48.318963 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:48.319368 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:48.319504 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:48.320099 (+ 595us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:15:48.320580 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:15:48.320765 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:48.320888 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 23:15:48.320932 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 23:15:48.329010 29679 raft_consensus.cc:1261] T 8555330edbe648af8040a82be14bb052 P bf3eaa6d12ea42f7ac04a96bb3603db3 [term 1 FOLLOWER]: Refusing update from remote peer 29d4991375dc4ecfa9f053a422317d76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:15:48.329012 29412 raft_consensus.cc:1261] T 8555330edbe648af8040a82be14bb052 P 758f92f6c27041d6906bb109cf8e13c6 [term 1 FOLLOWER]: Refusing update from remote peer 29d4991375dc4ecfa9f053a422317d76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:15:48.329993 29753 consensus_queue.cc:1035] T 8555330edbe648af8040a82be14bb052 P 29d4991375dc4ecfa9f053a422317d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "758f92f6c27041d6906bb109cf8e13c6" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 35775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:15:48.330322 29748 consensus_queue.cc:1035] T 8555330edbe648af8040a82be14bb052 P 29d4991375dc4ecfa9f053a422317d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf3eaa6d12ea42f7ac04a96bb3603db3" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 38499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:15:48.341102 29756 mvcc.cc:205] Tried to move back new op lower bound from 7299061138746011648 to 7299061138496659456. Current Snapshot: MvccSnapshot[applied={T|T < 7299061138746011648}]
I20260620 23:15:48.354337 29268 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:15:48.354615 29268 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:15:48.358384 29268 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c7f1456831c4f3abadfc59befeb950c: Sending DeleteTablet for 3 replicas of tablet 8555330edbe648af8040a82be14bb052
I20260620 23:15:48.359549 29392 tablet_service.cc:1514] Processing DeleteTablet for tablet 8555330edbe648af8040a82be14bb052 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:15:48 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:57770
I20260620 23:15:48.359555 29659 tablet_service.cc:1514] Processing DeleteTablet for tablet 8555330edbe648af8040a82be14bb052 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:15:48 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:36142
I20260620 23:15:48.360528 29767 tablet_replica.cc:330] T 8555330edbe648af8040a82be14bb052 P bf3eaa6d12ea42f7ac04a96bb3603db3: stopping tablet replica
I20260620 23:15:48.361095 29767 raft_consensus.cc:2229] T 8555330edbe648af8040a82be14bb052 P bf3eaa6d12ea42f7ac04a96bb3603db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:15:48.361778 29767 raft_consensus.cc:2258] T 8555330edbe648af8040a82be14bb052 P bf3eaa6d12ea42f7ac04a96bb3603db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:15:48.362758 29526 tablet_service.cc:1514] Processing DeleteTablet for tablet 8555330edbe648af8040a82be14bb052 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:15:48 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:48942
I20260620 23:15:48.364660 29767 ts_tablet_manager.cc:1905] T 8555330edbe648af8040a82be14bb052 P bf3eaa6d12ea42f7ac04a96bb3603db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:15:48.364773 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 29329
I20260620 23:15:48.372123 29766 tablet_replica.cc:330] T 8555330edbe648af8040a82be14bb052 P 758f92f6c27041d6906bb109cf8e13c6: stopping tablet replica
I20260620 23:15:48.372511 29766 raft_consensus.cc:2229] T 8555330edbe648af8040a82be14bb052 P 758f92f6c27041d6906bb109cf8e13c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:15:48.372954 29766 raft_consensus.cc:2258] T 8555330edbe648af8040a82be14bb052 P 758f92f6c27041d6906bb109cf8e13c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:15:48.374980 29766 ts_tablet_manager.cc:1905] T 8555330edbe648af8040a82be14bb052 P 758f92f6c27041d6906bb109cf8e13c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:15:48.375265 29767 ts_tablet_manager.cc:1918] T 8555330edbe648af8040a82be14bb052 P bf3eaa6d12ea42f7ac04a96bb3603db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:15:48.375435 29767 log.cc:1196] T 8555330edbe648af8040a82be14bb052 P bf3eaa6d12ea42f7ac04a96bb3603db3: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-2/wal/wals/8555330edbe648af8040a82be14bb052
I20260620 23:15:48.375932 29767 ts_tablet_manager.cc:1939] T 8555330edbe648af8040a82be14bb052 P bf3eaa6d12ea42f7ac04a96bb3603db3: Deleting consensus metadata
I20260620 23:15:48.377769 29255 catalog_manager.cc:4841] TS bf3eaa6d12ea42f7ac04a96bb3603db3 (127.10.191.195:38499): tablet 8555330edbe648af8040a82be14bb052 (table test-table [id=8405d9af4da4482d80c6130ea49dd7c4]) successfully deleted
I20260620 23:15:48.378036 29768 tablet_replica.cc:330] T 8555330edbe648af8040a82be14bb052 P 29d4991375dc4ecfa9f053a422317d76: stopping tablet replica
I20260620 23:15:48.378530 29768 raft_consensus.cc:2229] T 8555330edbe648af8040a82be14bb052 P 29d4991375dc4ecfa9f053a422317d76 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:15:48.379216 29768 raft_consensus.cc:2258] T 8555330edbe648af8040a82be14bb052 P 29d4991375dc4ecfa9f053a422317d76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:15:48.381392 29768 ts_tablet_manager.cc:1905] T 8555330edbe648af8040a82be14bb052 P 29d4991375dc4ecfa9f053a422317d76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:15:48.382332 29766 ts_tablet_manager.cc:1918] T 8555330edbe648af8040a82be14bb052 P 758f92f6c27041d6906bb109cf8e13c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:15:48.382495 29766 log.cc:1196] T 8555330edbe648af8040a82be14bb052 P 758f92f6c27041d6906bb109cf8e13c6: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-0/wal/wals/8555330edbe648af8040a82be14bb052
I20260620 23:15:48.383074 29766 ts_tablet_manager.cc:1939] T 8555330edbe648af8040a82be14bb052 P 758f92f6c27041d6906bb109cf8e13c6: Deleting consensus metadata
I20260620 23:15:48.384763 29253 catalog_manager.cc:4841] TS 758f92f6c27041d6906bb109cf8e13c6 (127.10.191.193:35775): tablet 8555330edbe648af8040a82be14bb052 (table test-table [id=8405d9af4da4482d80c6130ea49dd7c4]) successfully deleted
Jun 20 23:15:48 dist-test-slave-jkbq krb5kdc[29223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.193@KRBTEST.COM
I20260620 23:15:48.388685 29768 ts_tablet_manager.cc:1918] T 8555330edbe648af8040a82be14bb052 P 29d4991375dc4ecfa9f053a422317d76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:15:48.388841 29768 log.cc:1196] T 8555330edbe648af8040a82be14bb052 P 29d4991375dc4ecfa9f053a422317d76: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781997042893763-11007-0/minicluster-data/ts-1/wal/wals/8555330edbe648af8040a82be14bb052
I20260620 23:15:48.389408 29768 ts_tablet_manager.cc:1939] T 8555330edbe648af8040a82be14bb052 P 29d4991375dc4ecfa9f053a422317d76: Deleting consensus metadata
I20260620 23:15:48.391098 29255 catalog_manager.cc:4841] TS 29d4991375dc4ecfa9f053a422317d76 (127.10.191.194:46257): tablet 8555330edbe648af8040a82be14bb052 (table test-table [id=8405d9af4da4482d80c6130ea49dd7c4]) successfully deleted
I20260620 23:15:48.395644 29739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:48.365960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57782 (local address 127.10.191.193:35775)
0620 23:15:48.366149 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:48.366157 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:48.378345 (+ 12188us) server_negotiation.cc:407] Connection header received
0620 23:15:48.379242 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:48.379247 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:48.379371 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:48.379457 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:48.381296 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:48.382438 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:48.384542 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:48.385124 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:48.385210 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:48.389764 (+ 4554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:48.389794 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:48.389814 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:48.389857 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:48.392860 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:48.393532 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:48.393540 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:48.393544 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:48.393680 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:48.394048 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:48.394052 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:48.394056 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:48.394409 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:48.394706 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:48.395091 (+ 385us) server_negotiation.cc:299] Negotiation successful
0620 23:15:48.395296 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:15:48.396349 29452 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:15:48.502230 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 29329
I20260620 23:15:48.519428 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 29463
Jun 20 23:15:48 dist-test-slave-jkbq krb5kdc[29223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.194@KRBTEST.COM
I20260620 23:15:48.535617 29740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:48.520188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48956 (local address 127.10.191.194:46257)
0620 23:15:48.520378 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:48.520386 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:48.520407 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:15:48.520621 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:48.520625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:48.520696 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:48.520799 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:48.522087 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:48.522973 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:48.524227 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:48.524679 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:48.524825 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:48.529517 (+ 4692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:48.529533 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:48.529551 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:48.529598 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:48.532670 (+ 3072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:48.533435 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:48.533442 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:48.533444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:48.533578 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:48.533953 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:48.533956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:48.533958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:48.534300 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:48.534540 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:48.535076 (+ 536us) server_negotiation.cc:299] Negotiation successful
0620 23:15:48.535240 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:15:48.536396 29586 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:15:48.649716 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 29463
I20260620 23:15:48.668195 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 29596
Jun 20 23:15:48 dist-test-slave-jkbq krb5kdc[29223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.195@KRBTEST.COM
I20260620 23:15:48.682737 29738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:48.668897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36150 (local address 127.10.191.195:38499)
0620 23:15:48.669056 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:48.669060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:48.669077 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:15:48.669237 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:48.669241 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:48.669303 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:48.669397 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:48.670674 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:48.671440 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:48.672632 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:48.672991 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:48.673076 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:48.677614 (+ 4538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:48.677628 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:48.677643 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:48.677681 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:48.680386 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:48.681047 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:48.681054 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:48.681056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:48.681194 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:48.681524 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:48.681527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:48.681529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:48.681868 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:48.682068 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:48.682327 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 23:15:48.682481 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 23:15:48.683485 29719 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:15:48.772081 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 29596
I20260620 23:15:48.788815 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 29236
I20260620 23:15:48.789884 29298 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:15:48.900096 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 29236
2026-06-20T23:15:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4361 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T23:15:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:15:48Z Disabled control of system clock
I20260620 23:15:48.975440 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:40361
--webserver_interface=127.10.191.254
--webserver_port=0
--builtin_ntp_servers=127.10.191.212:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:40361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:15:49.294505 29777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:49.294775 29777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:49.294947 29777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:49.301934 29777 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:15:49.302023 29777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:49.302094 29777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:49.302213 29777 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:15:49.302301 29777 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:15:49.311089 29777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:40361
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:40361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:15:49.312304 29777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:49.314067 29777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:49.325677 29782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:49.325668 29783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:49.325703 29785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:49.326745 29777 server_base.cc:1034] running on GCE node
I20260620 23:15:49.327505 29777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:49.328896 29777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:49.330086 29777 hybrid_clock.cc:648] HybridClock initialized: now 1781997349330034 us; error 50 us; skew 500 ppm
I20260620 23:15:49.330444 29777 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:15:49.332711 29777 webserver.cc:460] Webserver started at http://127.10.191.254:38135/ using document root <none> and password file <none>
I20260620 23:15:49.333381 29777 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:49.333494 29777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:49.333823 29777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:15:49.336639 29777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "f4a64804c318442b81d2dd9b93176bdb"
format_stamp: "Formatted at 2026-06-20 23:15:49 on dist-test-slave-jkbq"
server_key: "c3570fc1871f0dc66b3d99c5f3c6b081"
server_key_iv: "d1ae43767657466e58fd40f9e50aa1e3"
server_key_version: "encryptionkey@0"
I20260620 23:15:49.337422 29777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "f4a64804c318442b81d2dd9b93176bdb"
format_stamp: "Formatted at 2026-06-20 23:15:49 on dist-test-slave-jkbq"
server_key: "c3570fc1871f0dc66b3d99c5f3c6b081"
server_key_iv: "d1ae43767657466e58fd40f9e50aa1e3"
server_key_version: "encryptionkey@0"
I20260620 23:15:49.342206 29777 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260620 23:15:49.345647 29791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:49.346985 29777 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 23:15:49.347160 29777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "f4a64804c318442b81d2dd9b93176bdb"
format_stamp: "Formatted at 2026-06-20 23:15:49 on dist-test-slave-jkbq"
server_key: "c3570fc1871f0dc66b3d99c5f3c6b081"
server_key_iv: "d1ae43767657466e58fd40f9e50aa1e3"
server_key_version: "encryptionkey@0"
I20260620 23:15:49.347378 29777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:49.376530 29777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:49.377339 29777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:49.377626 29777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:49.396126 29777 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:40361
I20260620 23:15:49.396209 29842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:40361 every 8 connection(s)
I20260620 23:15:49.397454 29777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:15:49.401212 29843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:15:49.405633 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 29777
I20260620 23:15:49.405819 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:15:49.406265 11007 external_mini_cluster.cc:1428] Setting key e97d25ebad3527ec4117b3efd9ec9aab
I20260620 23:15:49.412943 29843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4a64804c318442b81d2dd9b93176bdb: Bootstrap starting.
I20260620 23:15:49.415983 29843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4a64804c318442b81d2dd9b93176bdb: Neither blocks nor log segments found. Creating new log.
I20260620 23:15:49.416991 29843 log.cc:824] T 00000000000000000000000000000000 P f4a64804c318442b81d2dd9b93176bdb: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:49.419903 29843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4a64804c318442b81d2dd9b93176bdb: No bootstrap required, opened a new log
I20260620 23:15:49.420282 29846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:49.408624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60228 (local address 127.10.191.254:40361)
0620 23:15:49.409927 (+ 1303us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:49.409937 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:49.409982 (+ 45us) server_negotiation.cc:407] Connection header received
0620 23:15:49.411926 (+ 1944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:49.411946 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:49.412359 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:49.412722 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:49.413269 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:49.414547 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:49.415935 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:49.416673 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:49.416891 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:49.418038 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:49.418051 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:49.418070 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:15:49.418623 (+ 553us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:49.418694 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:15:49.418718 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:49.418800 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:49.419076 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 23:15:49.419372 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":814,"thread_start_us":579,"threads_started":1}
I20260620 23:15:49.427412 29843 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4a64804c318442b81d2dd9b93176bdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4a64804c318442b81d2dd9b93176bdb" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 40361 } }
I20260620 23:15:49.427750 29843 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4a64804c318442b81d2dd9b93176bdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:15:49.427851 29843 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4a64804c318442b81d2dd9b93176bdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4a64804c318442b81d2dd9b93176bdb, State: Initialized, Role: FOLLOWER
I20260620 23:15:49.428416 29843 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4a64804c318442b81d2dd9b93176bdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4a64804c318442b81d2dd9b93176bdb" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 40361 } }
I20260620 23:15:49.428722 29843 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4a64804c318442b81d2dd9b93176bdb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:15:49.428875 29843 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4a64804c318442b81d2dd9b93176bdb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:15:49.429069 29843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4a64804c318442b81d2dd9b93176bdb [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:15:49.431227 29843 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4a64804c318442b81d2dd9b93176bdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4a64804c318442b81d2dd9b93176bdb" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 40361 } }
I20260620 23:15:49.431710 29843 leader_election.cc:302] T 00000000000000000000000000000000 P f4a64804c318442b81d2dd9b93176bdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4a64804c318442b81d2dd9b93176bdb; no voters:
I20260620 23:15:49.432216 29843 leader_election.cc:288] T 00000000000000000000000000000000 P f4a64804c318442b81d2dd9b93176bdb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:15:49.432330 29848 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4a64804c318442b81d2dd9b93176bdb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:15:49.433046 29848 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4a64804c318442b81d2dd9b93176bdb [term 1 LEADER]: Becoming Leader. State: Replica: f4a64804c318442b81d2dd9b93176bdb, State: Running, Role: LEADER
I20260620 23:15:49.433506 29848 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4a64804c318442b81d2dd9b93176bdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4a64804c318442b81d2dd9b93176bdb" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 40361 } }
I20260620 23:15:49.434432 29843 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4a64804c318442b81d2dd9b93176bdb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:15:49.436440 29850 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4a64804c318442b81d2dd9b93176bdb [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4a64804c318442b81d2dd9b93176bdb. Latest consensus state: current_term: 1 leader_uuid: "f4a64804c318442b81d2dd9b93176bdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4a64804c318442b81d2dd9b93176bdb" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 40361 } } }
I20260620 23:15:49.436904 29850 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4a64804c318442b81d2dd9b93176bdb [sys.catalog]: This master's current role is: LEADER
I20260620 23:15:49.439021 29849 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4a64804c318442b81d2dd9b93176bdb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4a64804c318442b81d2dd9b93176bdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4a64804c318442b81d2dd9b93176bdb" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 40361 } } }
I20260620 23:15:49.439249 29849 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4a64804c318442b81d2dd9b93176bdb [sys.catalog]: This master's current role is: LEADER
I20260620 23:15:49.439340 29856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:15:49.444195 29856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:15:49.454819 29856 catalog_manager.cc:1347] Generated new cluster ID: 7ebd674efa244cb48a1177d797036e49
I20260620 23:15:49.454957 29856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:15:49.465368 29856 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:15:49.466564 29856 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:15:49.473192 29856 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4a64804c318442b81d2dd9b93176bdb: Generated new TSK 0
I20260620 23:15:49.473898 29856 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:15:49.481817 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.193:0
--local_ip_for_outbound_sockets=127.10.191.193
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:40361
--builtin_ntp_servers=127.10.191.212:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:15:49.806281 29867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:49.806592 29867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:49.806758 29867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:49.813851 29867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:49.813982 29867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:49.814158 29867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.193
I20260620 23:15:49.822937 29867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:40361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.193
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:15:49.824177 29867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:49.825976 29867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:49.838843 29872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:49.838922 29873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:49.838934 29875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:49.839748 29867 server_base.cc:1034] running on GCE node
I20260620 23:15:49.840364 29867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:49.841292 29867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:49.842530 29867 hybrid_clock.cc:648] HybridClock initialized: now 1781997349842467 us; error 53 us; skew 500 ppm
I20260620 23:15:49.842855 29867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:15:49.845253 29867 webserver.cc:460] Webserver started at http://127.10.191.193:38213/ using document root <none> and password file <none>
I20260620 23:15:49.845920 29867 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:49.846050 29867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:49.846364 29867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:15:49.849299 29867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-0/data/instance:
uuid: "0fd419a9960244f18fe3dfe8bdc8f636"
format_stamp: "Formatted at 2026-06-20 23:15:49 on dist-test-slave-jkbq"
server_key: "61b0ca4ed3fa08a1ae13d1dd229ee1ab"
server_key_iv: "54f6377d4ae0d2b1c92598c98096e335"
server_key_version: "encryptionkey@0"
I20260620 23:15:49.850066 29867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance:
uuid: "0fd419a9960244f18fe3dfe8bdc8f636"
format_stamp: "Formatted at 2026-06-20 23:15:49 on dist-test-slave-jkbq"
server_key: "61b0ca4ed3fa08a1ae13d1dd229ee1ab"
server_key_iv: "54f6377d4ae0d2b1c92598c98096e335"
server_key_version: "encryptionkey@0"
I20260620 23:15:49.855126 29867 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 23:15:49.858539 29881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:49.859964 29867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:15:49.860131 29867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-0/wal
uuid: "0fd419a9960244f18fe3dfe8bdc8f636"
format_stamp: "Formatted at 2026-06-20 23:15:49 on dist-test-slave-jkbq"
server_key: "61b0ca4ed3fa08a1ae13d1dd229ee1ab"
server_key_iv: "54f6377d4ae0d2b1c92598c98096e335"
server_key_version: "encryptionkey@0"
I20260620 23:15:49.860337 29867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:49.876108 29867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:49.876900 29867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:49.877202 29867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:49.878132 29867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:15:49.879608 29867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:15:49.879733 29867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:49.879865 29867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:15:49.879952 29867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:49.913367 29867 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.193:38715
I20260620 23:15:49.913412 29993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.193:38715 every 8 connection(s)
I20260620 23:15:49.914731 29867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
I20260620 23:15:49.919584 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 29867
I20260620 23:15:49.919792 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance
I20260620 23:15:49.920364 11007 external_mini_cluster.cc:1428] Setting key 4b9ae064f9d0228b8439fbf708b4cb81
I20260620 23:15:49.923552 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.194:0
--local_ip_for_outbound_sockets=127.10.191.194
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:40361
--builtin_ntp_servers=127.10.191.212:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 23:15:49.929215 29846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:49.917245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:40567 (local address 127.10.191.254:40361)
0620 23:15:49.917461 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:49.917468 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:49.918917 (+ 1449us) server_negotiation.cc:407] Connection header received
0620 23:15:49.921447 (+ 2530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:49.921453 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:49.921540 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:49.921627 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:49.923032 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:49.923822 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:49.926330 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:49.926722 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:49.926815 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:49.928442 (+ 1627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:49.928446 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:49.928449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:15:49.928466 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:49.928496 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:15:49.928508 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:49.928578 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:49.928873 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 23:15:49.928951 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 23:15:49.930655 29996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:49.917804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:40361 (local address 127.10.191.193:40567)
0620 23:15:49.918745 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:49.918782 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:49.921202 (+ 2420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:49.921906 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:49.921918 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:49.922037 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:49.922824 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:49.922851 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:49.923961 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:49.923968 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:49.926176 (+ 2208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:49.926194 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:49.926964 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:49.926974 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:49.927305 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:49.928068 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:15:49.928073 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:49.928131 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:15:49.928240 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:49.928603 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:49.928617 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:49.928627 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 23:15:49.928809 (+ 182us) client_negotiation.cc:240] Negotiation successful
0620 23:15:49.929256 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":296,"threads_started":1}
I20260620 23:15:49.932200 29994 heartbeater.cc:347] Connected to a master server at 127.10.191.254:40361
I20260620 23:15:49.932507 29994 heartbeater.cc:464] Registering TS with master...
I20260620 23:15:49.933264 29994 heartbeater.cc:511] Master 127.10.191.254:40361 requested a full tablet report, sending...
I20260620 23:15:49.935536 29808 ts_manager.cc:194] Registered new tserver with Master: 0fd419a9960244f18fe3dfe8bdc8f636 (127.10.191.193:38715)
I20260620 23:15:49.937770 29808 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.191.193:40567
W20260620 23:15:50.242808 29997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:50.243140 29997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:50.243235 29997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:50.250519 29997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:50.250612 29997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:50.250761 29997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.194
I20260620 23:15:50.259755 29997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:40361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.194
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:15:50.260924 29997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:50.262626 29997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:50.275595 30002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:50.275652 30003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:50.275676 30005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:50.276672 29997 server_base.cc:1034] running on GCE node
I20260620 23:15:50.277216 29997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:50.278100 29997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:50.279309 29997 hybrid_clock.cc:648] HybridClock initialized: now 1781997350279247 us; error 43 us; skew 500 ppm
I20260620 23:15:50.279618 29997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:15:50.284502 29997 webserver.cc:460] Webserver started at http://127.10.191.194:41725/ using document root <none> and password file <none>
I20260620 23:15:50.285434 29997 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:50.285554 29997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:50.285905 29997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:15:50.289891 29997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-1/data/instance:
uuid: "b9ddc3ff85a54bd5bbd8537b7a314f90"
format_stamp: "Formatted at 2026-06-20 23:15:50 on dist-test-slave-jkbq"
server_key: "88751016df1fbbc104b1c3348e72b656"
server_key_iv: "57f2297dc786ff409951a89b4a3029c9"
server_key_version: "encryptionkey@0"
I20260620 23:15:50.290657 29997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance:
uuid: "b9ddc3ff85a54bd5bbd8537b7a314f90"
format_stamp: "Formatted at 2026-06-20 23:15:50 on dist-test-slave-jkbq"
server_key: "88751016df1fbbc104b1c3348e72b656"
server_key_iv: "57f2297dc786ff409951a89b4a3029c9"
server_key_version: "encryptionkey@0"
I20260620 23:15:50.295543 29997 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260620 23:15:50.298794 30011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:50.300168 29997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:15:50.300338 29997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-1/wal
uuid: "b9ddc3ff85a54bd5bbd8537b7a314f90"
format_stamp: "Formatted at 2026-06-20 23:15:50 on dist-test-slave-jkbq"
server_key: "88751016df1fbbc104b1c3348e72b656"
server_key_iv: "57f2297dc786ff409951a89b4a3029c9"
server_key_version: "encryptionkey@0"
I20260620 23:15:50.300561 29997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:50.312235 29997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:50.312983 29997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:50.313282 29997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:50.314178 29997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:15:50.315615 29997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:15:50.315733 29997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:50.315865 29997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:15:50.315954 29997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:50.347935 29997 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.194:42221
I20260620 23:15:50.347987 30123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.194:42221 every 8 connection(s)
I20260620 23:15:50.349200 29997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
I20260620 23:15:50.350591 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 29997
I20260620 23:15:50.350775 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance
I20260620 23:15:50.351390 11007 external_mini_cluster.cc:1428] Setting key a25f3a3cf53591eb2e9be91ea4589c7c
I20260620 23:15:50.354274 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.195:0
--local_ip_for_outbound_sockets=127.10.191.195
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:40361
--builtin_ntp_servers=127.10.191.212:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 23:15:50.361785 29846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:50.351865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:36763 (local address 127.10.191.254:40361)
0620 23:15:50.352045 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:50.352050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:50.353703 (+ 1653us) server_negotiation.cc:407] Connection header received
0620 23:15:50.355597 (+ 1894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:50.355603 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:50.355684 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:50.355804 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:50.356809 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:50.357535 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:50.359115 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:50.359528 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:50.359668 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:50.360981 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:50.360987 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:50.360991 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:15:50.361017 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:50.361049 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:15:50.361061 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:50.361155 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:50.361379 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 23:15:50.361478 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 23:15:50.362735 30126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:50.352513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:40361 (local address 127.10.191.194:36763)
0620 23:15:50.353567 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:50.353603 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:50.355342 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:50.356029 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:50.356037 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:50.356128 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:50.356648 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:50.356664 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:50.357693 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:50.357700 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:50.358970 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:50.358982 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:50.359747 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:50.359754 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:50.359993 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:50.360680 (+ 687us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:15:50.360683 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:50.360722 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:15:50.360800 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:50.361153 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:50.361165 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:50.361176 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 23:15:50.361371 (+ 195us) client_negotiation.cc:240] Negotiation successful
0620 23:15:50.361732 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":330,"threads_started":1}
I20260620 23:15:50.364228 30124 heartbeater.cc:347] Connected to a master server at 127.10.191.254:40361
I20260620 23:15:50.364547 30124 heartbeater.cc:464] Registering TS with master...
I20260620 23:15:50.365288 30124 heartbeater.cc:511] Master 127.10.191.254:40361 requested a full tablet report, sending...
I20260620 23:15:50.366603 29808 ts_manager.cc:194] Registered new tserver with Master: b9ddc3ff85a54bd5bbd8537b7a314f90 (127.10.191.194:42221)
I20260620 23:15:50.367434 29808 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.191.194:36763
W20260620 23:15:50.671979 30127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:50.672278 30127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:50.672448 30127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:50.679631 30127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:50.679756 30127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:50.679934 30127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.195
I20260620 23:15:50.688999 30127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:40361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.195
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:15:50.690267 30127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:50.692005 30127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:50.704995 30132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:50.704960 30135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:50.705029 30133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:50.705576 30127 server_base.cc:1034] running on GCE node
I20260620 23:15:50.706147 30127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:50.707079 30127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:50.708273 30127 hybrid_clock.cc:648] HybridClock initialized: now 1781997350708234 us; error 29 us; skew 500 ppm
I20260620 23:15:50.708572 30127 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:15:50.716180 30127 webserver.cc:460] Webserver started at http://127.10.191.195:44929/ using document root <none> and password file <none>
I20260620 23:15:50.716849 30127 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:50.716936 30127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:50.717280 30127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:15:50.720093 30127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-2/data/instance:
uuid: "f6b488ed7c4345f1abc1491c274b5939"
format_stamp: "Formatted at 2026-06-20 23:15:50 on dist-test-slave-jkbq"
server_key: "3064aaafcb1e92fb81396f60f5f35ed7"
server_key_iv: "75633a7508e0a566e9e9e85f1786d670"
server_key_version: "encryptionkey@0"
I20260620 23:15:50.720855 30127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance:
uuid: "f6b488ed7c4345f1abc1491c274b5939"
format_stamp: "Formatted at 2026-06-20 23:15:50 on dist-test-slave-jkbq"
server_key: "3064aaafcb1e92fb81396f60f5f35ed7"
server_key_iv: "75633a7508e0a566e9e9e85f1786d670"
server_key_version: "encryptionkey@0"
I20260620 23:15:50.725898 30127 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 23:15:50.729189 30141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:50.730479 30127 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:15:50.730661 30127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-2/wal
uuid: "f6b488ed7c4345f1abc1491c274b5939"
format_stamp: "Formatted at 2026-06-20 23:15:50 on dist-test-slave-jkbq"
server_key: "3064aaafcb1e92fb81396f60f5f35ed7"
server_key_iv: "75633a7508e0a566e9e9e85f1786d670"
server_key_version: "encryptionkey@0"
I20260620 23:15:50.730870 30127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:50.755474 30127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:50.756587 30127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:50.756923 30127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:50.758242 30127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:15:50.760278 30127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:15:50.760397 30127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:50.760511 30127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:15:50.760593 30127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:50.796205 30127 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.195:43867
I20260620 23:15:50.796254 30253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.195:43867 every 8 connection(s)
I20260620 23:15:50.797523 30127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
I20260620 23:15:50.797916 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 30127
I20260620 23:15:50.798115 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance
I20260620 23:15:50.798671 11007 external_mini_cluster.cc:1428] Setting key 1a4e8085e134b8d1ab13454adfd974fd
I20260620 23:15:50.810053 29846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:50.800367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:45019 (local address 127.10.191.254:40361)
0620 23:15:50.800608 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:50.800613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:50.802189 (+ 1576us) server_negotiation.cc:407] Connection header received
0620 23:15:50.804047 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:50.804051 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:50.804140 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:50.804227 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:50.805342 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:50.806058 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:50.807629 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:50.808023 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:50.808122 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:50.809383 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:50.809390 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:50.809394 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:15:50.809421 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:50.809453 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:15:50.809465 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:50.809550 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:50.809746 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 23:15:50.809824 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260620 23:15:50.811079 30256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:50.800992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:40361 (local address 127.10.191.195:45019)
0620 23:15:50.802022 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:50.802066 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:50.803795 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:50.804430 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:50.804438 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:50.804557 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:50.805143 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:50.805159 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:50.806197 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:50.806200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:50.807501 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:50.807512 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:50.808198 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:50.808202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:50.808389 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:50.809090 (+ 701us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:15:50.809093 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:50.809130 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:15:50.809208 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:50.809553 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:50.809560 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:50.809565 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:15:50.809699 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 23:15:50.809984 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":343,"threads_started":1}
I20260620 23:15:50.812510 30254 heartbeater.cc:347] Connected to a master server at 127.10.191.254:40361
I20260620 23:15:50.812863 30254 heartbeater.cc:464] Registering TS with master...
I20260620 23:15:50.813803 30254 heartbeater.cc:511] Master 127.10.191.254:40361 requested a full tablet report, sending...
I20260620 23:15:50.815133 29808 ts_manager.cc:194] Registered new tserver with Master: f6b488ed7c4345f1abc1491c274b5939 (127.10.191.195:43867)
I20260620 23:15:50.815917 29808 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.191.195:45019
I20260620 23:15:50.825884 11007 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:15:50.835975 29846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:50.829881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60232 (local address 127.10.191.254:40361)
0620 23:15:50.830036 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:50.830040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:50.830490 (+ 450us) server_negotiation.cc:407] Connection header received
0620 23:15:50.830623 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:50.830627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:50.830722 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:50.830801 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:50.831271 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:50.832101 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:50.833436 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:50.833813 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:50.833900 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:50.835150 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:50.835155 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:50.835158 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:15:50.835174 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:50.835205 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:15:50.835216 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:50.835327 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:50.835560 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 23:15:50.835615 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 23:15:50.839561 29808 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:15:50.846267 29808 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:15:50.870615 30266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:50.858842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.194:42221 (local address 127.0.0.1:38362)
0620 23:15:50.860399 (+ 1557us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:50.860447 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:50.860615 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:50.862705 (+ 2090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:50.862712 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:50.862744 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:50.863167 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:50.863177 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:50.865362 (+ 2185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:50.865366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:50.867941 (+ 2575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:50.867954 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:50.869638 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:50.869644 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:50.869824 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:50.869869 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 23:15:50.869987 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 23:15:50.870201 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":201,"thread_start_us":347,"threads_started":1}
I20260620 23:15:50.871531 30268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:50.859739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35530 (local address 127.10.191.195:43867)
0620 23:15:50.860885 (+ 1146us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:50.860902 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:50.861617 (+ 715us) server_negotiation.cc:407] Connection header received
0620 23:15:50.861726 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:50.861733 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:50.862029 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:50.862271 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:50.863742 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:50.865231 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:50.868653 (+ 3422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:50.869517 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:50.869725 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:50.869953 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:50.870961 (+ 1008us) server_negotiation.cc:299] Negotiation successful
0620 23:15:50.871156 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":953,"thread_start_us":795,"threads_started":1}
I20260620 23:15:50.871668 30267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:50.859008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38362 (local address 127.10.191.194:42221)
0620 23:15:50.860208 (+ 1200us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:50.860216 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:50.862097 (+ 1881us) server_negotiation.cc:407] Connection header received
0620 23:15:50.862222 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:50.862230 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:50.862481 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:50.863382 (+ 901us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:50.863631 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:50.865197 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:50.868134 (+ 2937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:50.869457 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:50.869645 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:50.869863 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:50.871177 (+ 1314us) server_negotiation.cc:299] Negotiation successful
0620 23:15:50.871335 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1021,"thread_start_us":900,"threads_started":1}
I20260620 23:15:50.872079 30270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:50.859678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.195:43867 (local address 127.0.0.1:35530)
0620 23:15:50.861282 (+ 1604us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:50.861306 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:50.861435 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:50.862275 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:50.862282 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:50.862311 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:50.862829 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:50.862845 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:50.865389 (+ 2544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:50.865393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:50.868411 (+ 3018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:50.868426 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:50.870012 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:50.870019 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:50.870227 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:50.870258 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 23:15:50.870347 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 23:15:50.870485 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1475,"mutex_wait_us":393,"thread_start_us":2163,"threads_started":1}
I20260620 23:15:50.872110 30269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:50.860726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:38715 (local address 127.0.0.1:51190)
0620 23:15:50.862220 (+ 1494us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:50.862243 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:50.862352 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:50.863350 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:50.863357 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:50.863388 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:50.863809 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:50.863838 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:50.865691 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:50.865696 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:50.868411 (+ 2715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:50.868426 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:50.870208 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:50.870213 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:50.870382 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:50.870412 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 23:15:50.870504 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 23:15:50.870642 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1352,"mutex_wait_us":233,"thread_start_us":316,"threads_started":1}
I20260620 23:15:50.871574 30271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:50.861231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51190 (local address 127.10.191.193:38715)
0620 23:15:50.861919 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:50.861928 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:50.862592 (+ 664us) server_negotiation.cc:407] Connection header received
0620 23:15:50.862824 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:50.862831 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:50.863059 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:50.863248 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:50.864125 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:50.865525 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:50.868626 (+ 3101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:50.870012 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:50.870187 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:50.870395 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:50.871071 (+ 676us) server_negotiation.cc:299] Negotiation successful
0620 23:15:50.871236 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":349,"threads_started":1}
I20260620 23:15:50.875646 29929 tablet_service.cc:1467] Processing CreateTablet for tablet f16cea18541349b698cd6edf2a1336a8 (DEFAULT_TABLE table=test-table [id=4ca94890c75e4ae2b8d1a283f6c8a5c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:15:50.875648 30059 tablet_service.cc:1467] Processing CreateTablet for tablet f16cea18541349b698cd6edf2a1336a8 (DEFAULT_TABLE table=test-table [id=4ca94890c75e4ae2b8d1a283f6c8a5c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:15:50.876317 30189 tablet_service.cc:1467] Processing CreateTablet for tablet f16cea18541349b698cd6edf2a1336a8 (DEFAULT_TABLE table=test-table [id=4ca94890c75e4ae2b8d1a283f6c8a5c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:15:50.877090 29929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f16cea18541349b698cd6edf2a1336a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:15:50.877089 30059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f16cea18541349b698cd6edf2a1336a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:15:50.877518 30189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f16cea18541349b698cd6edf2a1336a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:15:50.889202 30272 tablet_bootstrap.cc:492] T f16cea18541349b698cd6edf2a1336a8 P f6b488ed7c4345f1abc1491c274b5939: Bootstrap starting.
I20260620 23:15:50.891515 30273 tablet_bootstrap.cc:492] T f16cea18541349b698cd6edf2a1336a8 P b9ddc3ff85a54bd5bbd8537b7a314f90: Bootstrap starting.
I20260620 23:15:50.893038 30272 tablet_bootstrap.cc:654] T f16cea18541349b698cd6edf2a1336a8 P f6b488ed7c4345f1abc1491c274b5939: Neither blocks nor log segments found. Creating new log.
I20260620 23:15:50.894295 30272 log.cc:824] T f16cea18541349b698cd6edf2a1336a8 P f6b488ed7c4345f1abc1491c274b5939: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:50.895109 30273 tablet_bootstrap.cc:654] T f16cea18541349b698cd6edf2a1336a8 P b9ddc3ff85a54bd5bbd8537b7a314f90: Neither blocks nor log segments found. Creating new log.
I20260620 23:15:50.895840 30274 tablet_bootstrap.cc:492] T f16cea18541349b698cd6edf2a1336a8 P 0fd419a9960244f18fe3dfe8bdc8f636: Bootstrap starting.
I20260620 23:15:50.896389 30273 log.cc:824] T f16cea18541349b698cd6edf2a1336a8 P b9ddc3ff85a54bd5bbd8537b7a314f90: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:50.897756 30272 tablet_bootstrap.cc:492] T f16cea18541349b698cd6edf2a1336a8 P f6b488ed7c4345f1abc1491c274b5939: No bootstrap required, opened a new log
I20260620 23:15:50.898015 30272 ts_tablet_manager.cc:1397] T f16cea18541349b698cd6edf2a1336a8 P f6b488ed7c4345f1abc1491c274b5939: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 23:15:50.899803 30274 tablet_bootstrap.cc:654] T f16cea18541349b698cd6edf2a1336a8 P 0fd419a9960244f18fe3dfe8bdc8f636: Neither blocks nor log segments found. Creating new log.
I20260620 23:15:50.901255 30274 log.cc:824] T f16cea18541349b698cd6edf2a1336a8 P 0fd419a9960244f18fe3dfe8bdc8f636: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:50.904237 30274 tablet_bootstrap.cc:492] T f16cea18541349b698cd6edf2a1336a8 P 0fd419a9960244f18fe3dfe8bdc8f636: No bootstrap required, opened a new log
I20260620 23:15:50.904524 30274 ts_tablet_manager.cc:1397] T f16cea18541349b698cd6edf2a1336a8 P 0fd419a9960244f18fe3dfe8bdc8f636: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 23:15:50.904610 30273 tablet_bootstrap.cc:492] T f16cea18541349b698cd6edf2a1336a8 P b9ddc3ff85a54bd5bbd8537b7a314f90: No bootstrap required, opened a new log
I20260620 23:15:50.904858 30273 ts_tablet_manager.cc:1397] T f16cea18541349b698cd6edf2a1336a8 P b9ddc3ff85a54bd5bbd8537b7a314f90: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.012s
I20260620 23:15:50.907086 30272 raft_consensus.cc:348] T f16cea18541349b698cd6edf2a1336a8 P f6b488ed7c4345f1abc1491c274b5939 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9ddc3ff85a54bd5bbd8537b7a314f90" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 42221 } } peers { permanent_uuid: "0fd419a9960244f18fe3dfe8bdc8f636" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 38715 } } peers { permanent_uuid: "f6b488ed7c4345f1abc1491c274b5939" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 43867 } }
I20260620 23:15:50.907500 30272 raft_consensus.cc:374] T f16cea18541349b698cd6edf2a1336a8 P f6b488ed7c4345f1abc1491c274b5939 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:15:50.907600 30272 raft_consensus.cc:729] T f16cea18541349b698cd6edf2a1336a8 P f6b488ed7c4345f1abc1491c274b5939 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b488ed7c4345f1abc1491c274b5939, State: Initialized, Role: FOLLOWER
I20260620 23:15:50.908195 30272 consensus_queue.cc:260] T f16cea18541349b698cd6edf2a1336a8 P f6b488ed7c4345f1abc1491c274b5939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9ddc3ff85a54bd5bbd8537b7a314f90" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 42221 } } peers { permanent_uuid: "0fd419a9960244f18fe3dfe8bdc8f636" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 38715 } } peers { permanent_uuid: "f6b488ed7c4345f1abc1491c274b5939" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 43867 } }
I20260620 23:15:50.911587 30254 heartbeater.cc:503] Master 127.10.191.254:40361 was elected leader, sending a full tablet report...
I20260620 23:15:50.912218 30272 ts_tablet_manager.cc:1428] T f16cea18541349b698cd6edf2a1336a8 P f6b488ed7c4345f1abc1491c274b5939: Time spent starting tablet: real 0.014s user 0.007s sys 0.007s
I20260620 23:15:50.913233 30274 raft_consensus.cc:348] T f16cea18541349b698cd6edf2a1336a8 P 0fd419a9960244f18fe3dfe8bdc8f636 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9ddc3ff85a54bd5bbd8537b7a314f90" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 42221 } } peers { permanent_uuid: "0fd419a9960244f18fe3dfe8bdc8f636" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 38715 } } peers { permanent_uuid: "f6b488ed7c4345f1abc1491c274b5939" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 43867 } }
I20260620 23:15:50.913682 30274 raft_consensus.cc:374] T f16cea18541349b698cd6edf2a1336a8 P 0fd419a9960244f18fe3dfe8bdc8f636 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:15:50.913784 30274 raft_consensus.cc:729] T f16cea18541349b698cd6edf2a1336a8 P 0fd419a9960244f18fe3dfe8bdc8f636 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fd419a9960244f18fe3dfe8bdc8f636, State: Initialized, Role: FOLLOWER
I20260620 23:15:50.914513 30274 consensus_queue.cc:260] T f16cea18541349b698cd6edf2a1336a8 P 0fd419a9960244f18fe3dfe8bdc8f636 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9ddc3ff85a54bd5bbd8537b7a314f90" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 42221 } } peers { permanent_uuid: "0fd419a9960244f18fe3dfe8bdc8f636" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 38715 } } peers { permanent_uuid: "f6b488ed7c4345f1abc1491c274b5939" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 43867 } }
I20260620 23:15:50.915771 30273 raft_consensus.cc:348] T f16cea18541349b698cd6edf2a1336a8 P b9ddc3ff85a54bd5bbd8537b7a314f90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9ddc3ff85a54bd5bbd8537b7a314f90" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 42221 } } peers { permanent_uuid: "0fd419a9960244f18fe3dfe8bdc8f636" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 38715 } } peers { permanent_uuid: "f6b488ed7c4345f1abc1491c274b5939" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 43867 } }
I20260620 23:15:50.916227 30273 raft_consensus.cc:374] T f16cea18541349b698cd6edf2a1336a8 P b9ddc3ff85a54bd5bbd8537b7a314f90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:15:50.916360 30273 raft_consensus.cc:729] T f16cea18541349b698cd6edf2a1336a8 P b9ddc3ff85a54bd5bbd8537b7a314f90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9ddc3ff85a54bd5bbd8537b7a314f90, State: Initialized, Role: FOLLOWER
I20260620 23:15:50.917110 30273 consensus_queue.cc:260] T f16cea18541349b698cd6edf2a1336a8 P b9ddc3ff85a54bd5bbd8537b7a314f90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9ddc3ff85a54bd5bbd8537b7a314f90" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 42221 } } peers { permanent_uuid: "0fd419a9960244f18fe3dfe8bdc8f636" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 38715 } } peers { permanent_uuid: "f6b488ed7c4345f1abc1491c274b5939" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 43867 } }
I20260620 23:15:50.921717 30124 heartbeater.cc:503] Master 127.10.191.254:40361 was elected leader, sending a full tablet report...
I20260620 23:15:50.922271 30273 ts_tablet_manager.cc:1428] T f16cea18541349b698cd6edf2a1336a8 P b9ddc3ff85a54bd5bbd8537b7a314f90: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20260620 23:15:50.922641 30274 ts_tablet_manager.cc:1428] T f16cea18541349b698cd6edf2a1336a8 P 0fd419a9960244f18fe3dfe8bdc8f636: Time spent starting tablet: real 0.018s user 0.014s sys 0.004s
I20260620 23:15:50.923027 29994 heartbeater.cc:503] Master 127.10.191.254:40361 was elected leader, sending a full tablet report...
W20260620 23:15:51.049716 30255 tablet.cc:2369] T f16cea18541349b698cd6edf2a1336a8 P f6b488ed7c4345f1abc1491c274b5939: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:15:51.101419 30125 tablet.cc:2369] T f16cea18541349b698cd6edf2a1336a8 P b9ddc3ff85a54bd5bbd8537b7a314f90: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:15:51.149799 30280 raft_consensus.cc:482] T f16cea18541349b698cd6edf2a1336a8 P b9ddc3ff85a54bd5bbd8537b7a314f90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:15:51.150152 30280 raft_consensus.cc:504] T f16cea18541349b698cd6edf2a1336a8 P b9ddc3ff85a54bd5bbd8537b7a314f90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9ddc3ff85a54bd5bbd8537b7a314f90" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 42221 } } peers { permanent_uuid: "0fd419a9960244f18fe3dfe8bdc8f636" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 38715 } } peers { permanent_uuid: "f6b488ed7c4345f1abc1491c274b5939" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 43867 } }
I20260620 23:15:51.152052 30280 leader_election.cc:288] T f16cea18541349b698cd6edf2a1336a8 P b9ddc3ff85a54bd5bbd8537b7a314f90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fd419a9960244f18fe3dfe8bdc8f636 (127.10.191.193:38715), f6b488ed7c4345f1abc1491c274b5939 (127.10.191.195:43867)
I20260620 23:15:51.159181 30268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:51.152505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:54851 (local address 127.10.191.195:43867)
0620 23:15:51.152704 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:51.152711 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:51.153493 (+ 782us) server_negotiation.cc:407] Connection header received
0620 23:15:51.153698 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:51.153704 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:51.153786 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:51.153889 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:51.154828 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:51.155596 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:51.157466 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:51.158209 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:51.158396 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:51.158460 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:51.158811 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 23:15:51.158949 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 23:15:51.159186 30271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:51.152576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:36701 (local address 127.10.191.193:38715)
0620 23:15:51.152788 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:51.152796 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:51.153497 (+ 701us) server_negotiation.cc:407] Connection header received
0620 23:15:51.153698 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:51.153704 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:51.153793 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:51.153896 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:51.154785 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:51.155519 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:51.157434 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:51.158188 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:51.158334 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:51.158424 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:51.158765 (+ 341us) server_negotiation.cc:299] Negotiation successful
0620 23:15:51.158917 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 23:15:51.159265 30281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:51.152424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:38715 (local address 127.10.191.194:36701)
0620 23:15:51.153383 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:51.153401 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:51.153550 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:51.153919 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:51.153924 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:51.153958 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:51.154586 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:51.154608 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:51.155663 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:51.155670 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:51.157232 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:51.157244 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:51.158407 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:51.158414 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:51.158610 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:51.158639 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 23:15:51.158708 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 23:15:51.158845 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":861,"mutex_wait_us":116,"spinlock_wait_cycles":26496,"thread_start_us":522,"threads_started":1}
I20260620 23:15:51.159324 30282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:51.152325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.195:43867 (local address 127.10.191.194:54851)
0620 23:15:51.153343 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:51.153366 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:51.153550 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:51.153919 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:51.153924 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:51.153958 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:51.154664 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:51.154677 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:51.155734 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:51.155738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:51.157246 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:51.157258 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:51.158424 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:51.158431 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:51.158634 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:51.158657 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 23:15:51.158763 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 23:15:51.158920 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":870,"mutex_wait_us":113,"spinlock_wait_cycles":20096,"thread_start_us":689,"threads_started":1}
I20260620 23:15:51.160413 29949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f16cea18541349b698cd6edf2a1336a8" candidate_uuid: "b9ddc3ff85a54bd5bbd8537b7a314f90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fd419a9960244f18fe3dfe8bdc8f636" is_pre_election: true
I20260620 23:15:51.160578 30209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f16cea18541349b698cd6edf2a1336a8" candidate_uuid: "b9ddc3ff85a54bd5bbd8537b7a314f90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b488ed7c4345f1abc1491c274b5939" is_pre_election: true
I20260620 23:15:51.160928 29949 raft_consensus.cc:2454] T f16cea18541349b698cd6edf2a1336a8 P 0fd419a9960244f18fe3dfe8bdc8f636 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9ddc3ff85a54bd5bbd8537b7a314f90 in term 0.
I20260620 23:15:51.161088 30209 raft_consensus.cc:2454] T f16cea18541349b698cd6edf2a1336a8 P f6b488ed7c4345f1abc1491c274b5939 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9ddc3ff85a54bd5bbd8537b7a314f90 in term 0.
I20260620 23:15:51.161602 30012 leader_election.cc:302] T f16cea18541349b698cd6edf2a1336a8 P b9ddc3ff85a54bd5bbd8537b7a314f90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fd419a9960244f18fe3dfe8bdc8f636, b9ddc3ff85a54bd5bbd8537b7a314f90; no voters:
I20260620 23:15:51.161968 30280 raft_consensus.cc:2781] T f16cea18541349b698cd6edf2a1336a8 P b9ddc3ff85a54bd5bbd8537b7a314f90 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:15:51.162108 30280 raft_consensus.cc:482] T f16cea18541349b698cd6edf2a1336a8 P b9ddc3ff85a54bd5bbd8537b7a314f90 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:15:51.162217 30280 raft_consensus.cc:3037] T f16cea18541349b698cd6edf2a1336a8 P b9ddc3ff85a54bd5bbd8537b7a314f90 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:15:51.164641 30280 raft_consensus.cc:504] T f16cea18541349b698cd6edf2a1336a8 P b9ddc3ff85a54bd5bbd8537b7a314f90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9ddc3ff85a54bd5bbd8537b7a314f90" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 42221 } } peers { permanent_uuid: "0fd419a9960244f18fe3dfe8bdc8f636" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 38715 } } peers { permanent_uuid: "f6b488ed7c4345f1abc1491c274b5939" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 43867 } }
I20260620 23:15:51.165189 30280 leader_election.cc:288] T f16cea18541349b698cd6edf2a1336a8 P b9ddc3ff85a54bd5bbd8537b7a314f90 [CANDIDATE]: Term 1 election: Requested vote from peers 0fd419a9960244f18fe3dfe8bdc8f636 (127.10.191.193:38715), f6b488ed7c4345f1abc1491c274b5939 (127.10.191.195:43867)
I20260620 23:15:51.165570 29949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f16cea18541349b698cd6edf2a1336a8" candidate_uuid: "b9ddc3ff85a54bd5bbd8537b7a314f90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fd419a9960244f18fe3dfe8bdc8f636"
I20260620 23:15:51.165611 30209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f16cea18541349b698cd6edf2a1336a8" candidate_uuid: "b9ddc3ff85a54bd5bbd8537b7a314f90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b488ed7c4345f1abc1491c274b5939"
I20260620 23:15:51.165791 30209 raft_consensus.cc:3037] T f16cea18541349b698cd6edf2a1336a8 P f6b488ed7c4345f1abc1491c274b5939 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:15:51.165781 29949 raft_consensus.cc:3037] T f16cea18541349b698cd6edf2a1336a8 P 0fd419a9960244f18fe3dfe8bdc8f636 [term 0 FOLLOWER]: Advancing to term 1
W20260620 23:15:51.167104 29995 tablet.cc:2369] T f16cea18541349b698cd6edf2a1336a8 P 0fd419a9960244f18fe3dfe8bdc8f636: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:15:51.168957 29949 raft_consensus.cc:2454] T f16cea18541349b698cd6edf2a1336a8 P 0fd419a9960244f18fe3dfe8bdc8f636 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9ddc3ff85a54bd5bbd8537b7a314f90 in term 1.
I20260620 23:15:51.168957 30209 raft_consensus.cc:2454] T f16cea18541349b698cd6edf2a1336a8 P f6b488ed7c4345f1abc1491c274b5939 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9ddc3ff85a54bd5bbd8537b7a314f90 in term 1.
I20260620 23:15:51.169368 30013 leader_election.cc:302] T f16cea18541349b698cd6edf2a1336a8 P b9ddc3ff85a54bd5bbd8537b7a314f90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9ddc3ff85a54bd5bbd8537b7a314f90, f6b488ed7c4345f1abc1491c274b5939; no voters:
I20260620 23:15:51.169632 30280 raft_consensus.cc:2781] T f16cea18541349b698cd6edf2a1336a8 P b9ddc3ff85a54bd5bbd8537b7a314f90 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:15:51.170065 30280 raft_consensus.cc:686] T f16cea18541349b698cd6edf2a1336a8 P b9ddc3ff85a54bd5bbd8537b7a314f90 [term 1 LEADER]: Becoming Leader. State: Replica: b9ddc3ff85a54bd5bbd8537b7a314f90, State: Running, Role: LEADER
I20260620 23:15:51.170506 30280 consensus_queue.cc:237] T f16cea18541349b698cd6edf2a1336a8 P b9ddc3ff85a54bd5bbd8537b7a314f90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9ddc3ff85a54bd5bbd8537b7a314f90" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 42221 } } peers { permanent_uuid: "0fd419a9960244f18fe3dfe8bdc8f636" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 38715 } } peers { permanent_uuid: "f6b488ed7c4345f1abc1491c274b5939" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 43867 } }
I20260620 23:15:51.174988 29808 catalog_manager.cc:5495] T f16cea18541349b698cd6edf2a1336a8 P b9ddc3ff85a54bd5bbd8537b7a314f90 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9ddc3ff85a54bd5bbd8537b7a314f90 (127.10.191.194). New cstate: current_term: 1 leader_uuid: "b9ddc3ff85a54bd5bbd8537b7a314f90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9ddc3ff85a54bd5bbd8537b7a314f90" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 42221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fd419a9960244f18fe3dfe8bdc8f636" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 38715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6b488ed7c4345f1abc1491c274b5939" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 43867 } health_report { overall_health: UNKNOWN } } }
I20260620 23:15:51.246115 30267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:51.240823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38376 (local address 127.10.191.194:42221)
0620 23:15:51.241069 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:51.241073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:51.241092 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:15:51.241280 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:51.241284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:51.241368 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:51.241488 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:15:51.241903 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:51.242591 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:51.244030 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:51.244440 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:51.244532 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:51.245020 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:15:51.245496 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:15:51.245617 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:51.245770 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 23:15:51.245832 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260620 23:15:51.254588 30209 raft_consensus.cc:1261] T f16cea18541349b698cd6edf2a1336a8 P f6b488ed7c4345f1abc1491c274b5939 [term 1 FOLLOWER]: Refusing update from remote peer b9ddc3ff85a54bd5bbd8537b7a314f90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:15:51.254587 29949 raft_consensus.cc:1261] T f16cea18541349b698cd6edf2a1336a8 P 0fd419a9960244f18fe3dfe8bdc8f636 [term 1 FOLLOWER]: Refusing update from remote peer b9ddc3ff85a54bd5bbd8537b7a314f90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:15:51.255676 30280 consensus_queue.cc:1035] T f16cea18541349b698cd6edf2a1336a8 P b9ddc3ff85a54bd5bbd8537b7a314f90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fd419a9960244f18fe3dfe8bdc8f636" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:15:51.255959 30283 consensus_queue.cc:1035] T f16cea18541349b698cd6edf2a1336a8 P b9ddc3ff85a54bd5bbd8537b7a314f90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b488ed7c4345f1abc1491c274b5939" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 43867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:15:51.267266 30286 mvcc.cc:205] Tried to move back new op lower bound from 7299061150728396800 to 7299061150397894656. Current Snapshot: MvccSnapshot[applied={T|T < 7299061150728396800}]
I20260620 23:15:51.272078 30288 mvcc.cc:205] Tried to move back new op lower bound from 7299061150728396800 to 7299061150397894656. Current Snapshot: MvccSnapshot[applied={T|T < 7299061150728396800}]
I20260620 23:15:51.281165 29806 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:15:51.281431 29806 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:15:51.284888 29806 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4a64804c318442b81d2dd9b93176bdb: Sending DeleteTablet for 3 replicas of tablet f16cea18541349b698cd6edf2a1336a8
I20260620 23:15:51.285997 30059 tablet_service.cc:1514] Processing DeleteTablet for tablet f16cea18541349b698cd6edf2a1336a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:15:51 UTC) from {username='slave'} at 127.0.0.1:38362
I20260620 23:15:51.286018 29929 tablet_service.cc:1514] Processing DeleteTablet for tablet f16cea18541349b698cd6edf2a1336a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:15:51 UTC) from {username='slave'} at 127.0.0.1:51190
I20260620 23:15:51.286972 30296 tablet_replica.cc:330] T f16cea18541349b698cd6edf2a1336a8 P b9ddc3ff85a54bd5bbd8537b7a314f90: stopping tablet replica
I20260620 23:15:51.286873 30189 tablet_service.cc:1514] Processing DeleteTablet for tablet f16cea18541349b698cd6edf2a1336a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:15:51 UTC) from {username='slave'} at 127.0.0.1:35530
I20260620 23:15:51.287461 30296 raft_consensus.cc:2229] T f16cea18541349b698cd6edf2a1336a8 P b9ddc3ff85a54bd5bbd8537b7a314f90 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:15:51.288175 30296 raft_consensus.cc:2258] T f16cea18541349b698cd6edf2a1336a8 P b9ddc3ff85a54bd5bbd8537b7a314f90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:15:51.288673 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 29867
I20260620 23:15:51.291896 30296 ts_tablet_manager.cc:1905] T f16cea18541349b698cd6edf2a1336a8 P b9ddc3ff85a54bd5bbd8537b7a314f90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:15:51.301339 30298 tablet_replica.cc:330] T f16cea18541349b698cd6edf2a1336a8 P f6b488ed7c4345f1abc1491c274b5939: stopping tablet replica
I20260620 23:15:51.301966 30298 raft_consensus.cc:2229] T f16cea18541349b698cd6edf2a1336a8 P f6b488ed7c4345f1abc1491c274b5939 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:15:51.302677 30298 raft_consensus.cc:2258] T f16cea18541349b698cd6edf2a1336a8 P f6b488ed7c4345f1abc1491c274b5939 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:15:51.303158 30296 ts_tablet_manager.cc:1918] T f16cea18541349b698cd6edf2a1336a8 P b9ddc3ff85a54bd5bbd8537b7a314f90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:15:51.303414 30296 log.cc:1196] T f16cea18541349b698cd6edf2a1336a8 P b9ddc3ff85a54bd5bbd8537b7a314f90: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-1/wal/wals/f16cea18541349b698cd6edf2a1336a8
I20260620 23:15:51.304242 30296 ts_tablet_manager.cc:1939] T f16cea18541349b698cd6edf2a1336a8 P b9ddc3ff85a54bd5bbd8537b7a314f90: Deleting consensus metadata
I20260620 23:15:51.306133 29792 catalog_manager.cc:4841] TS b9ddc3ff85a54bd5bbd8537b7a314f90 (127.10.191.194:42221): tablet f16cea18541349b698cd6edf2a1336a8 (table test-table [id=4ca94890c75e4ae2b8d1a283f6c8a5c0]) successfully deleted
I20260620 23:15:51.306488 30298 ts_tablet_manager.cc:1905] T f16cea18541349b698cd6edf2a1336a8 P f6b488ed7c4345f1abc1491c274b5939: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:15:51.307255 30297 tablet_replica.cc:330] T f16cea18541349b698cd6edf2a1336a8 P 0fd419a9960244f18fe3dfe8bdc8f636: stopping tablet replica
I20260620 23:15:51.307685 30297 raft_consensus.cc:2229] T f16cea18541349b698cd6edf2a1336a8 P 0fd419a9960244f18fe3dfe8bdc8f636 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:15:51.308203 30297 raft_consensus.cc:2258] T f16cea18541349b698cd6edf2a1336a8 P 0fd419a9960244f18fe3dfe8bdc8f636 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:15:51.310438 30297 ts_tablet_manager.cc:1905] T f16cea18541349b698cd6edf2a1336a8 P 0fd419a9960244f18fe3dfe8bdc8f636: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:15:51.310828 30271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:51.289611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51206 (local address 127.10.191.193:38715)
0620 23:15:51.289795 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:51.289802 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:51.295229 (+ 5427us) server_negotiation.cc:407] Connection header received
0620 23:15:51.295554 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:51.295563 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:51.295694 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:51.295785 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:51.296585 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:51.298196 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:51.303743 (+ 5547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:51.304570 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:51.304723 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:51.309120 (+ 4397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:51.309137 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:51.309159 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:15:51.309913 (+ 754us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:51.309981 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:15:51.310006 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:51.310088 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:51.310333 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 23:15:51.310470 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 23:15:51.312366 29989 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:15:51.313118 30298 ts_tablet_manager.cc:1918] T f16cea18541349b698cd6edf2a1336a8 P f6b488ed7c4345f1abc1491c274b5939: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:15:51.313405 30298 log.cc:1196] T f16cea18541349b698cd6edf2a1336a8 P f6b488ed7c4345f1abc1491c274b5939: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781997042893763-11007-0/minicluster-data/ts-2/wal/wals/f16cea18541349b698cd6edf2a1336a8
I20260620 23:15:51.314205 30298 ts_tablet_manager.cc:1939] T f16cea18541349b698cd6edf2a1336a8 P f6b488ed7c4345f1abc1491c274b5939: Deleting consensus metadata
I20260620 23:15:51.316797 29793 catalog_manager.cc:4841] TS f6b488ed7c4345f1abc1491c274b5939 (127.10.191.195:43867): tablet f16cea18541349b698cd6edf2a1336a8 (table test-table [id=4ca94890c75e4ae2b8d1a283f6c8a5c0]) successfully deleted
I20260620 23:15:51.463742 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 29867
W20260620 23:15:51.481168 29792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.191.193:38715 (error 108)
I20260620 23:15:51.481367 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 29997
I20260620 23:15:51.486040 30302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:51.483958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:38715 (local address 127.0.0.1:51210)
0620 23:15:51.485683 (+ 1725us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:51.485830 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.193:38715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1591,"thread_start_us":1515,"threads_started":1}
W20260620 23:15:51.486318 29792 catalog_manager.cc:4584] TS 0fd419a9960244f18fe3dfe8bdc8f636 (127.10.191.193:38715): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f16cea18541349b698cd6edf2a1336a8: Network error: Client connection negotiation failed: client connection to 127.10.191.193:38715: connect: Connection refused (error 111)
I20260620 23:15:51.490110 30267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:51.482168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38386 (local address 127.10.191.194:42221)
0620 23:15:51.482795 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:51.482802 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:51.482819 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:15:51.482901 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:51.482907 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:51.483003 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:51.483114 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:51.483674 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:51.485184 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:51.487120 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:51.487549 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:51.487648 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:51.488754 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:51.488762 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:51.488783 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:15:51.489385 (+ 602us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:51.489430 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:15:51.489455 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:51.489532 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:51.489692 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 23:15:51.489764 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494}
I20260620 23:15:51.490810 30119 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:15:51.539846 30302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:51.539338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:38715 (local address 127.0.0.1:51226)
0620 23:15:51.539562 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:51.539672 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.193:38715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 23:15:51.540971 30302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:51.540549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:38715 (local address 127.0.0.1:51230)
0620 23:15:51.540719 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:51.540804 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.193:38715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 23:15:51.601831 30302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:51.601351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:38715 (local address 127.0.0.1:51240)
0620 23:15:51.601557 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:51.601678 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.193:38715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 23:15:51.602447 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 29997
I20260620 23:15:51.604841 30302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:51.604465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:38715 (local address 127.0.0.1:51242)
0620 23:15:51.604607 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:51.604709 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.193:38715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 23:15:51.620952 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 30127
I20260620 23:15:51.627849 30268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:51.621732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35542 (local address 127.10.191.195:43867)
0620 23:15:51.621916 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:51.621921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:51.621984 (+ 63us) server_negotiation.cc:407] Connection header received
0620 23:15:51.622216 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:51.622222 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:51.622320 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:51.622433 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:51.622820 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:51.623580 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:51.624863 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:51.625268 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:51.625406 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:51.626441 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:51.626450 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:51.626464 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:15:51.627102 (+ 638us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:51.627149 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:15:51.627170 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:51.627289 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:51.627428 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 23:15:51.627532 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:15:51.628440 30249 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:15:51.685258 30302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:51.684678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:38715 (local address 127.0.0.1:51248)
0620 23:15:51.684955 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:51.685065 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.193:38715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260620 23:15:51.686419 30302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:51.685944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:38715 (local address 127.0.0.1:51256)
0620 23:15:51.686172 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:51.686253 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.193:38715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260620 23:15:51.715468 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 30127
I20260620 23:15:51.731511 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 29777
I20260620 23:15:51.732577 29838 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:15:51.844944 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 29777
2026-06-20T23:15:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2943 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:15:51 dist-test-slave-jkbq krb5kdc[30308](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:15:51 dist-test-slave-jkbq krb5kdc[30308](info): set up 2 sockets
Jun 20 23:15:51 dist-test-slave-jkbq krb5kdc[30308](info): commencing operation
krb5kdc: starting...
W20260620 23:15:52.959201 11007 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Jun 20 23:15:53 dist-test-slave-jkbq krb5kdc[30308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:15:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:15:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.191.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
I20260620 23:15:53.189592 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:37527
--webserver_interface=127.10.191.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:34937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:37527
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:15:53.510056 30321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:53.510313 30321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:53.510489 30321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:53.517554 30321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:15:53.517653 30321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:53.517750 30321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:53.517850 30321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:15:53.517932 30321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:15:53.527438 30321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:34937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:37527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:37527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:15:53.528680 30321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:53.529958 30321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:53.541615 30326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:53.541723 30327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:53.541755 30329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:53.542807 30321 server_base.cc:1034] running on GCE node
I20260620 23:15:53.543596 30321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:53.544868 30321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:53.546096 30321 hybrid_clock.cc:648] HybridClock initialized: now 1781997353546038 us; error 44 us; skew 500 ppm
I20260620 23:15:53.546418 30321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:15:53 dist-test-slave-jkbq krb5kdc[30308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997353, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:53.554478 30321 init.cc:377] Logged in from keytab as kudu/127.10.191.254@KRBTEST.COM (short username kudu)
I20260620 23:15:53.556007 30321 webserver.cc:460] Webserver started at http://127.10.191.254:39495/ using document root <none> and password file <none>
I20260620 23:15:53.556680 30321 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:53.556824 30321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:53.557130 30321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:15:53.560046 30321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "82436260b6e84e0a8959d79b5aa4dde2"
format_stamp: "Formatted at 2026-06-20 23:15:53 on dist-test-slave-jkbq"
server_key: "157e3661d2d2c7d876133588c2a24f2d"
server_key_iv: "1975d13a1912e5dce4dca4a64bbbadd3"
server_key_version: "encryptionkey@0"
I20260620 23:15:53.560801 30321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "82436260b6e84e0a8959d79b5aa4dde2"
format_stamp: "Formatted at 2026-06-20 23:15:53 on dist-test-slave-jkbq"
server_key: "157e3661d2d2c7d876133588c2a24f2d"
server_key_iv: "1975d13a1912e5dce4dca4a64bbbadd3"
server_key_version: "encryptionkey@0"
I20260620 23:15:53.566025 30321 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 23:15:53.569623 30336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:53.571007 30321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:15:53.571195 30321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "82436260b6e84e0a8959d79b5aa4dde2"
format_stamp: "Formatted at 2026-06-20 23:15:53 on dist-test-slave-jkbq"
server_key: "157e3661d2d2c7d876133588c2a24f2d"
server_key_iv: "1975d13a1912e5dce4dca4a64bbbadd3"
server_key_version: "encryptionkey@0"
I20260620 23:15:53.571399 30321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:53.596067 30321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:53.600188 30321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:53.600493 30321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:53.618927 30321 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:37527
I20260620 23:15:53.618952 30387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:37527 every 8 connection(s)
I20260620 23:15:53.620327 30321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:15:53.622172 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 30321
I20260620 23:15:53.622368 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:15:53.622996 11007 external_mini_cluster.cc:1428] Setting key 3f541c4bf8f8edf25c391fa2e8886507
I20260620 23:15:53.624783 30388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:15:53.635900 30388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82436260b6e84e0a8959d79b5aa4dde2: Bootstrap starting.
Jun 20 23:15:53 dist-test-slave-jkbq krb5kdc[30308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:15:53.639320 30388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82436260b6e84e0a8959d79b5aa4dde2: Neither blocks nor log segments found. Creating new log.
I20260620 23:15:53.640369 30388 log.cc:824] T 00000000000000000000000000000000 P 82436260b6e84e0a8959d79b5aa4dde2: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:53.643163 30388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82436260b6e84e0a8959d79b5aa4dde2: No bootstrap required, opened a new log
I20260620 23:15:53.644804 30391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:53.624915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58650 (local address 127.10.191.254:37527)
0620 23:15:53.625644 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:53.625656 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:53.625696 (+ 40us) server_negotiation.cc:407] Connection header received
0620 23:15:53.627378 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:53.627400 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:53.627779 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:53.628199 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:53.629698 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:53.631113 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:53.632428 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:53.633166 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:53.633383 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:53.637702 (+ 4319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:53.637710 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:53.637724 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:53.637763 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:53.640915 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:53.641614 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:53.641626 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:53.641632 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:53.641788 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:53.642177 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:53.642184 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:53.642187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:53.642707 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:53.643015 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:53.643465 (+ 450us) server_negotiation.cc:299] Negotiation successful
0620 23:15:53.643881 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":327,"threads_started":1}
I20260620 23:15:53.650828 30388 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82436260b6e84e0a8959d79b5aa4dde2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82436260b6e84e0a8959d79b5aa4dde2" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 37527 } }
I20260620 23:15:53.651263 30388 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82436260b6e84e0a8959d79b5aa4dde2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:15:53.651374 30388 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82436260b6e84e0a8959d79b5aa4dde2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82436260b6e84e0a8959d79b5aa4dde2, State: Initialized, Role: FOLLOWER
I20260620 23:15:53.652022 30388 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82436260b6e84e0a8959d79b5aa4dde2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82436260b6e84e0a8959d79b5aa4dde2" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 37527 } }
I20260620 23:15:53.652285 30388 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82436260b6e84e0a8959d79b5aa4dde2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:15:53.652436 30388 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82436260b6e84e0a8959d79b5aa4dde2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:15:53.652647 30388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82436260b6e84e0a8959d79b5aa4dde2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:15:53.654788 30388 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82436260b6e84e0a8959d79b5aa4dde2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82436260b6e84e0a8959d79b5aa4dde2" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 37527 } }
I20260620 23:15:53.655376 30388 leader_election.cc:302] T 00000000000000000000000000000000 P 82436260b6e84e0a8959d79b5aa4dde2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82436260b6e84e0a8959d79b5aa4dde2; no voters:
I20260620 23:15:53.655943 30388 leader_election.cc:288] T 00000000000000000000000000000000 P 82436260b6e84e0a8959d79b5aa4dde2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:15:53.656069 30393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82436260b6e84e0a8959d79b5aa4dde2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:15:53.656771 30393 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82436260b6e84e0a8959d79b5aa4dde2 [term 1 LEADER]: Becoming Leader. State: Replica: 82436260b6e84e0a8959d79b5aa4dde2, State: Running, Role: LEADER
I20260620 23:15:53.657384 30393 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82436260b6e84e0a8959d79b5aa4dde2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82436260b6e84e0a8959d79b5aa4dde2" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 37527 } }
I20260620 23:15:53.657989 30388 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82436260b6e84e0a8959d79b5aa4dde2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:15:53.660010 30395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82436260b6e84e0a8959d79b5aa4dde2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82436260b6e84e0a8959d79b5aa4dde2. Latest consensus state: current_term: 1 leader_uuid: "82436260b6e84e0a8959d79b5aa4dde2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82436260b6e84e0a8959d79b5aa4dde2" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 37527 } } }
I20260620 23:15:53.660393 30395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82436260b6e84e0a8959d79b5aa4dde2 [sys.catalog]: This master's current role is: LEADER
I20260620 23:15:53.660140 30394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82436260b6e84e0a8959d79b5aa4dde2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82436260b6e84e0a8959d79b5aa4dde2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82436260b6e84e0a8959d79b5aa4dde2" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 37527 } } }
I20260620 23:15:53.660540 30394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82436260b6e84e0a8959d79b5aa4dde2 [sys.catalog]: This master's current role is: LEADER
I20260620 23:15:53.662976 30398 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:15:53.667954 30398 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:15:53.676584 30398 catalog_manager.cc:1347] Generated new cluster ID: 4e8af8fb036742b697b006df9ff5aa97
I20260620 23:15:53.676697 30398 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:15:53.712608 30398 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:15:53.713917 30398 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:15:53.720552 30398 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82436260b6e84e0a8959d79b5aa4dde2: Generated new TSK 0
I20260620 23:15:53.721333 30398 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.191.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab.
Entry for principal kudu/127.10.191.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab.
I20260620 23:15:53.780031 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.193:0
--local_ip_for_outbound_sockets=127.10.191.193
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:37527
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:34937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:15:54.102312 30414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:54.102579 30414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:54.102741 30414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:54.110003 30414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:54.110105 30414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:54.110318 30414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.193
I20260620 23:15:54.120031 30414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:34937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.193.keytab
--principal=kudu/127.10.191.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:37527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.193
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:15:54.121469 30414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:54.122833 30414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:54.135973 30422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:54.136059 30419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:54.136094 30420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:54.137032 30414 server_base.cc:1034] running on GCE node
I20260620 23:15:54.137722 30414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:54.138672 30414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:54.139874 30414 hybrid_clock.cc:648] HybridClock initialized: now 1781997354139804 us; error 66 us; skew 500 ppm
I20260620 23:15:54.140196 30414 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:15:54 dist-test-slave-jkbq krb5kdc[30308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997354, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:54.143544 30414 init.cc:377] Logged in from keytab as kudu/127.10.191.193@KRBTEST.COM (short username kudu)
I20260620 23:15:54.145053 30414 webserver.cc:460] Webserver started at http://127.10.191.193:46305/ using document root <none> and password file <none>
I20260620 23:15:54.145732 30414 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:54.145862 30414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:54.146184 30414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:15:54.149267 30414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-0/data/instance:
uuid: "196905551f77460cb894f77322217edb"
format_stamp: "Formatted at 2026-06-20 23:15:54 on dist-test-slave-jkbq"
server_key: "560a50f61e3b658fa68a0a2a38b46351"
server_key_iv: "d136cf9a3191a04f0bb9b4ea4bec9676"
server_key_version: "encryptionkey@0"
I20260620 23:15:54.149973 30414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance:
uuid: "196905551f77460cb894f77322217edb"
format_stamp: "Formatted at 2026-06-20 23:15:54 on dist-test-slave-jkbq"
server_key: "560a50f61e3b658fa68a0a2a38b46351"
server_key_iv: "d136cf9a3191a04f0bb9b4ea4bec9676"
server_key_version: "encryptionkey@0"
I20260620 23:15:54.155203 30414 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 23:15:54.159161 30429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:54.160560 30414 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:15:54.160734 30414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-0/wal
uuid: "196905551f77460cb894f77322217edb"
format_stamp: "Formatted at 2026-06-20 23:15:54 on dist-test-slave-jkbq"
server_key: "560a50f61e3b658fa68a0a2a38b46351"
server_key_iv: "d136cf9a3191a04f0bb9b4ea4bec9676"
server_key_version: "encryptionkey@0"
I20260620 23:15:54.160949 30414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:54.177148 30414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:54.180941 30414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:54.181244 30414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:54.182221 30414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:15:54.183625 30414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:15:54.183727 30414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:54.183871 30414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:15:54.183952 30414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:54.217005 30414 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.193:46709
I20260620 23:15:54.217051 30541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.193:46709 every 8 connection(s)
I20260620 23:15:54.218309 30414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
I20260620 23:15:54.222828 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 30414
I20260620 23:15:54.223038 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance
I20260620 23:15:54.223587 11007 external_mini_cluster.cc:1428] Setting key 7c207adc34114fa58ca02000129e497b
Jun 20 23:15:54 dist-test-slave-jkbq krb5kdc[30308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997354, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.193@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:15:54.242074 30544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:54.220916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:55333 (local address 127.10.191.254:37527)
0620 23:15:54.221310 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:54.221315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:54.222333 (+ 1018us) server_negotiation.cc:407] Connection header received
0620 23:15:54.224812 (+ 2479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:54.224818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:54.224918 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:54.225036 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:54.227028 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:54.227751 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:54.230214 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:54.230628 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:54.230741 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:54.234903 (+ 4162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:54.234912 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:54.234917 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:54.234962 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:54.238528 (+ 3566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:54.239481 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:54.239490 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:54.239496 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:54.239642 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:54.240050 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:54.240057 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:54.240063 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:54.240461 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:54.240614 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:54.241351 (+ 737us) server_negotiation.cc:299] Negotiation successful
0620 23:15:54.241601 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":179,"threads_started":1}
I20260620 23:15:54.242794 30545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:54.221435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:37527 (local address 127.10.191.193:55333)
0620 23:15:54.222155 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:54.222195 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:54.224542 (+ 2347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:54.225284 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:54.225293 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:54.225998 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:54.226809 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:54.226836 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:54.227913 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:54.227921 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:54.230045 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:54.230063 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:54.230886 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:54.230895 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:54.231232 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:54.231985 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:15:54.232019 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:54.234642 (+ 2623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:54.238678 (+ 4036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:54.238686 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:54.238697 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:54.239348 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:54.239752 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:54.239755 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:54.239757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:54.239936 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:54.240588 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:54.240593 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:54.240965 (+ 372us) client_negotiation.cc:769] Sending connection context
0620 23:15:54.241193 (+ 228us) client_negotiation.cc:240] Negotiation successful
0620 23:15:54.241574 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":272,"threads_started":1}
I20260620 23:15:54.244443 30542 heartbeater.cc:347] Connected to a master server at 127.10.191.254:37527
I20260620 23:15:54.244751 30542 heartbeater.cc:464] Registering TS with master...
I20260620 23:15:54.245458 30542 heartbeater.cc:511] Master 127.10.191.254:37527 requested a full tablet report, sending...
I20260620 23:15:54.247582 30353 ts_manager.cc:194] Registered new tserver with Master: 196905551f77460cb894f77322217edb (127.10.191.193:46709)
I20260620 23:15:54.249010 30353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.193@KRBTEST.COM'} at 127.10.191.193:55333
WARNING: no policy specified for kudu/127.10.191.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab.
Entry for principal kudu/127.10.191.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab.
I20260620 23:15:54.282958 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.194:0
--local_ip_for_outbound_sockets=127.10.191.194
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:37527
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:34937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:15:54.603982 30548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:54.604238 30548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:54.604396 30548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:54.611467 30548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:54.611567 30548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:54.611774 30548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.194
I20260620 23:15:54.623440 30548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:34937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.194.keytab
--principal=kudu/127.10.191.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:37527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.194
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:15:54.624927 30548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:54.626435 30548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:54.641036 30556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:54.641121 30554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:54.641152 30553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:54.642081 30548 server_base.cc:1034] running on GCE node
I20260620 23:15:54.642660 30548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:54.643627 30548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:54.644865 30548 hybrid_clock.cc:648] HybridClock initialized: now 1781997354644813 us; error 46 us; skew 500 ppm
I20260620 23:15:54.645182 30548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:15:54 dist-test-slave-jkbq krb5kdc[30308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997354, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:54.648294 30548 init.cc:377] Logged in from keytab as kudu/127.10.191.194@KRBTEST.COM (short username kudu)
I20260620 23:15:54.649741 30548 webserver.cc:460] Webserver started at http://127.10.191.194:39813/ using document root <none> and password file <none>
I20260620 23:15:54.650414 30548 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:54.650553 30548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:54.650903 30548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:15:54.653674 30548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-1/data/instance:
uuid: "4d35210098d240a4ac37dfcb735c820e"
format_stamp: "Formatted at 2026-06-20 23:15:54 on dist-test-slave-jkbq"
server_key: "b258576093fc56f3d1e95b541138833b"
server_key_iv: "88390cd4596a4ff58a2a4c63e2de0f50"
server_key_version: "encryptionkey@0"
I20260620 23:15:54.654423 30548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance:
uuid: "4d35210098d240a4ac37dfcb735c820e"
format_stamp: "Formatted at 2026-06-20 23:15:54 on dist-test-slave-jkbq"
server_key: "b258576093fc56f3d1e95b541138833b"
server_key_iv: "88390cd4596a4ff58a2a4c63e2de0f50"
server_key_version: "encryptionkey@0"
I20260620 23:15:54.659631 30548 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260620 23:15:54.663174 30563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:54.664566 30548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:15:54.664722 30548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-1/wal
uuid: "4d35210098d240a4ac37dfcb735c820e"
format_stamp: "Formatted at 2026-06-20 23:15:54 on dist-test-slave-jkbq"
server_key: "b258576093fc56f3d1e95b541138833b"
server_key_iv: "88390cd4596a4ff58a2a4c63e2de0f50"
server_key_version: "encryptionkey@0"
I20260620 23:15:54.664923 30548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:54.684041 30548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:54.687992 30548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:54.688273 30548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:54.689221 30548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:15:54.690482 30548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:15:54.690582 30548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:54.690730 30548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:15:54.690800 30548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:54.723811 30548 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.194:36035
I20260620 23:15:54.723862 30675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.194:36035 every 8 connection(s)
I20260620 23:15:54.725096 30548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
I20260620 23:15:54.727290 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 30548
I20260620 23:15:54.727490 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance
I20260620 23:15:54.728070 11007 external_mini_cluster.cc:1428] Setting key 98727d4ab9d67cd9fbc3717e3b12a911
Jun 20 23:15:54 dist-test-slave-jkbq krb5kdc[30308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997354, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.194@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:15:54.747592 30544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:54.727989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:39307 (local address 127.10.191.254:37527)
0620 23:15:54.728226 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:54.728233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:54.729459 (+ 1226us) server_negotiation.cc:407] Connection header received
0620 23:15:54.731451 (+ 1992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:54.731460 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:54.731549 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:54.731657 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:54.733094 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:54.734183 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:54.735810 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:54.736575 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:54.736713 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:54.740685 (+ 3972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:54.740694 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:54.740701 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:54.740745 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:54.744284 (+ 3539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:54.745045 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:54.745053 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:54.745059 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:54.745231 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:54.745622 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:54.745630 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:54.745636 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:54.746007 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:54.746167 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:54.746814 (+ 647us) server_negotiation.cc:299] Negotiation successful
0620 23:15:54.747128 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 23:15:54.748242 30678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:54.728447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:37527 (local address 127.10.191.194:39307)
0620 23:15:54.729274 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:54.729323 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:54.731171 (+ 1848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:54.731843 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:54.731850 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:54.732371 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:54.732928 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:54.732946 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:54.734338 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:54.734342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:54.735666 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:54.735678 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:54.736789 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:54.736794 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:54.736994 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:54.737735 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:15:54.737776 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:54.740478 (+ 2702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:54.744451 (+ 3973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:54.744458 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:54.744470 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:54.744910 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:54.745338 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:54.745341 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:54.745343 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:54.745509 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:54.746121 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:54.746127 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:54.746434 (+ 307us) client_negotiation.cc:769] Sending connection context
0620 23:15:54.746656 (+ 222us) client_negotiation.cc:240] Negotiation successful
0620 23:15:54.747139 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":357,"threads_started":1}
I20260620 23:15:54.749773 30676 heartbeater.cc:347] Connected to a master server at 127.10.191.254:37527
I20260620 23:15:54.750097 30676 heartbeater.cc:464] Registering TS with master...
I20260620 23:15:54.751048 30676 heartbeater.cc:511] Master 127.10.191.254:37527 requested a full tablet report, sending...
I20260620 23:15:54.752770 30353 ts_manager.cc:194] Registered new tserver with Master: 4d35210098d240a4ac37dfcb735c820e (127.10.191.194:36035)
I20260620 23:15:54.753958 30353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.194@KRBTEST.COM'} at 127.10.191.194:39307
WARNING: no policy specified for kudu/127.10.191.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab.
Entry for principal kudu/127.10.191.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab.
I20260620 23:15:54.785805 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.195:0
--local_ip_for_outbound_sockets=127.10.191.195
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:37527
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:34937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:15:55.105084 30681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:55.105413 30681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:55.105588 30681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:55.112577 30681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:55.112726 30681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:55.112915 30681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.195
I20260620 23:15:55.122184 30681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:34937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.195.keytab
--principal=kudu/127.10.191.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:37527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.195
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:15:55.123471 30681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:55.124787 30681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:55.137866 30686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:55.137841 30689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:55.137830 30687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:55.139003 30681 server_base.cc:1034] running on GCE node
I20260620 23:15:55.139609 30681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:55.140632 30681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:55.141892 30681 hybrid_clock.cc:648] HybridClock initialized: now 1781997355141839 us; error 51 us; skew 500 ppm
I20260620 23:15:55.142237 30681 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:15:55 dist-test-slave-jkbq krb5kdc[30308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997355, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:15:55.145340 30681 init.cc:377] Logged in from keytab as kudu/127.10.191.195@KRBTEST.COM (short username kudu)
I20260620 23:15:55.146700 30681 webserver.cc:460] Webserver started at http://127.10.191.195:41497/ using document root <none> and password file <none>
I20260620 23:15:55.147394 30681 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:55.147537 30681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:55.147828 30681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:15:55.150563 30681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-2/data/instance:
uuid: "80f054cb213f481b90a85eb372010da4"
format_stamp: "Formatted at 2026-06-20 23:15:55 on dist-test-slave-jkbq"
server_key: "0324abf853a25c0882398d44f232637f"
server_key_iv: "e77df89c0e87da116aafc10edadf6f70"
server_key_version: "encryptionkey@0"
I20260620 23:15:55.151325 30681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance:
uuid: "80f054cb213f481b90a85eb372010da4"
format_stamp: "Formatted at 2026-06-20 23:15:55 on dist-test-slave-jkbq"
server_key: "0324abf853a25c0882398d44f232637f"
server_key_iv: "e77df89c0e87da116aafc10edadf6f70"
server_key_version: "encryptionkey@0"
I20260620 23:15:55.156231 30681 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 23:15:55.159893 30696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:55.161314 30681 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:15:55.161501 30681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-2/wal
uuid: "80f054cb213f481b90a85eb372010da4"
format_stamp: "Formatted at 2026-06-20 23:15:55 on dist-test-slave-jkbq"
server_key: "0324abf853a25c0882398d44f232637f"
server_key_iv: "e77df89c0e87da116aafc10edadf6f70"
server_key_version: "encryptionkey@0"
I20260620 23:15:55.161711 30681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:55.182323 30681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:55.186595 30681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:55.186926 30681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:55.187857 30681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:15:55.189218 30681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:15:55.189327 30681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:55.189471 30681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:15:55.189544 30681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:55.222886 30681 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.195:37015
I20260620 23:15:55.222934 30808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.195:37015 every 8 connection(s)
I20260620 23:15:55.224411 30681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
I20260620 23:15:55.229197 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 30681
I20260620 23:15:55.229365 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance
I20260620 23:15:55.229820 11007 external_mini_cluster.cc:1428] Setting key 290e81d279887622a813a76ed8184955
Jun 20 23:15:55 dist-test-slave-jkbq krb5kdc[30308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997355, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.195@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:15:55.243759 30544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:55.226961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:52055 (local address 127.10.191.254:37527)
0620 23:15:55.227150 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:55.227154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:55.228302 (+ 1148us) server_negotiation.cc:407] Connection header received
0620 23:15:55.230232 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:55.230240 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:55.230375 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:55.230491 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:55.232375 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:55.233151 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:55.234593 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:55.235049 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:55.235140 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:55.238328 (+ 3188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:55.238333 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:55.238336 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:55.238371 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:55.240748 (+ 2377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:55.241439 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:55.241446 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:55.241449 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:55.241573 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:55.241971 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:55.241977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:55.241980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:55.242264 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:55.242434 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:55.243144 (+ 710us) server_negotiation.cc:299] Negotiation successful
0620 23:15:55.243379 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:15:55.244824 30811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:55.227420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:37527 (local address 127.10.191.195:52055)
0620 23:15:55.228133 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:55.228170 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:55.229970 (+ 1800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:55.230771 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:55.230780 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:55.231557 (+ 777us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:55.232171 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:55.232188 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:55.233286 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:55.233290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:55.234475 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:55.234486 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:55.235225 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:55.235229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:55.235419 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:55.236074 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:15:55.236098 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:55.238137 (+ 2039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:55.240883 (+ 2746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:55.240891 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:55.240905 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:55.241320 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:55.241679 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:15:55.241684 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:15:55.241688 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:15:55.241863 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:15:55.242397 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:55.242405 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:55.242713 (+ 308us) client_negotiation.cc:769] Sending connection context
0620 23:15:55.243024 (+ 311us) client_negotiation.cc:240] Negotiation successful
0620 23:15:55.243519 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":272,"threads_started":1}
I20260620 23:15:55.246116 30809 heartbeater.cc:347] Connected to a master server at 127.10.191.254:37527
I20260620 23:15:55.246404 30809 heartbeater.cc:464] Registering TS with master...
I20260620 23:15:55.247143 30809 heartbeater.cc:511] Master 127.10.191.254:37527 requested a full tablet report, sending...
I20260620 23:15:55.248400 30353 ts_manager.cc:194] Registered new tserver with Master: 80f054cb213f481b90a85eb372010da4 (127.10.191.195:37015)
I20260620 23:15:55.249203 30353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.191.195@KRBTEST.COM'} at 127.10.191.195:52055
I20260620 23:15:55.252046 30542 heartbeater.cc:503] Master 127.10.191.254:37527 was elected leader, sending a full tablet report...
I20260620 23:15:55.256781 11007 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:15:55.273141 30544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:55.260757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58662 (local address 127.10.191.254:37527)
0620 23:15:55.260928 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:55.260932 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:55.261304 (+ 372us) server_negotiation.cc:407] Connection header received
0620 23:15:55.261435 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:55.261439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:55.261495 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:55.261575 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:55.263165 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:55.263914 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:55.265272 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:55.265763 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:55.265862 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:55.268316 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:55.268320 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:55.268323 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:55.268353 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:55.270987 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:55.271572 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:55.271576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:55.271578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:55.271698 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:55.272008 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:55.272011 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:55.272012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:55.272308 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:55.272480 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:55.272758 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 23:15:55.272911 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 23:15:55.276867 30353 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58662:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:15:55.283766 30353 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:15:55.305742 30821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:55.295192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:46709 (local address 127.0.0.1:34224)
0620 23:15:55.296010 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:55.296063 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:55.296239 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:55.297782 (+ 1543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:55.297793 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:55.297835 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:55.298373 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:55.298394 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:55.300733 (+ 2339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:55.300741 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:55.303263 (+ 2522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:55.303279 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:55.304868 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:55.304875 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:55.305074 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:55.305132 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 23:15:55.305229 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 23:15:55.305415 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"mutex_wait_us":61,"thread_start_us":350,"threads_started":1}
I20260620 23:15:55.306627 30824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:55.296532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51508 (local address 127.10.191.195:37015)
0620 23:15:55.297403 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:55.297415 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:55.298388 (+ 973us) server_negotiation.cc:407] Connection header received
0620 23:15:55.298678 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:55.298686 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:55.298959 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:55.299131 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:55.299977 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:55.301538 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:55.303856 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:55.305213 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:55.305394 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:55.305559 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:55.306101 (+ 542us) server_negotiation.cc:299] Negotiation successful
0620 23:15:55.306269 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":469,"threads_started":1}
I20260620 23:15:55.307039 30822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:55.295400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34224 (local address 127.10.191.193:46709)
0620 23:15:55.297020 (+ 1620us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:55.297032 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:55.297061 (+ 29us) server_negotiation.cc:407] Connection header received
0620 23:15:55.297160 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:55.297168 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:55.297495 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:55.298630 (+ 1135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:55.298938 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:55.300619 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:55.303432 (+ 2813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:55.304651 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:55.306302 (+ 1651us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:55.306472 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:55.306538 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 23:15:55.306693 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1416,"thread_start_us":1261,"threads_started":1}
I20260620 23:15:55.307076 30823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:55.297546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.194:36035 (local address 127.0.0.1:45646)
0620 23:15:55.299351 (+ 1805us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:55.299368 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:55.299536 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:55.300060 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:55.300066 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:55.300089 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:55.300387 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:55.300396 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:55.302211 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:55.302217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:55.304307 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:55.304320 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:55.306270 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:55.306279 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:55.306431 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:55.306459 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 23:15:55.306553 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 23:15:55.306673 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1652,"mutex_wait_us":57,"thread_start_us":456,"threads_started":1}
I20260620 23:15:55.307119 30825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:55.296822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45646 (local address 127.10.191.194:36035)
0620 23:15:55.297811 (+ 989us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:55.297819 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:55.299493 (+ 1674us) server_negotiation.cc:407] Connection header received
0620 23:15:55.299702 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:55.299708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:55.299856 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:55.300061 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:55.300555 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:55.302053 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:55.304654 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:55.306066 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:55.306217 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:55.306391 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:55.306609 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 23:15:55.306784 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":832,"thread_start_us":850,"threads_started":1}
I20260620 23:15:55.307598 30826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:55.295928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.195:37015 (local address 127.0.0.1:51508)
0620 23:15:55.298272 (+ 2344us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:55.298290 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:55.298489 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:55.299354 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:55.299359 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:55.299382 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:55.299776 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:55.299787 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:55.301691 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:55.301697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:55.303691 (+ 1994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:55.303707 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:55.305713 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:55.305720 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:55.305895 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:55.305927 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 23:15:55.307068 (+ 1141us) client_negotiation.cc:240] Negotiation successful
0620 23:15:55.307206 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2221,"mutex_wait_us":102,"thread_start_us":3148,"threads_started":1}
I20260620 23:15:55.310801 30744 tablet_service.cc:1467] Processing CreateTablet for tablet 32e965bf154d4035b26a1ec02b14df6d (DEFAULT_TABLE table=test-table [id=96b5c162c326437f860b012d9fd78aad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:15:55.310904 30611 tablet_service.cc:1467] Processing CreateTablet for tablet 32e965bf154d4035b26a1ec02b14df6d (DEFAULT_TABLE table=test-table [id=96b5c162c326437f860b012d9fd78aad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:15:55.310904 30477 tablet_service.cc:1467] Processing CreateTablet for tablet 32e965bf154d4035b26a1ec02b14df6d (DEFAULT_TABLE table=test-table [id=96b5c162c326437f860b012d9fd78aad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:15:55.312070 30744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32e965bf154d4035b26a1ec02b14df6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:15:55.312156 30477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32e965bf154d4035b26a1ec02b14df6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:15:55.312177 30611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32e965bf154d4035b26a1ec02b14df6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:15:55.322798 30827 tablet_bootstrap.cc:492] T 32e965bf154d4035b26a1ec02b14df6d P 80f054cb213f481b90a85eb372010da4: Bootstrap starting.
I20260620 23:15:55.325994 30827 tablet_bootstrap.cc:654] T 32e965bf154d4035b26a1ec02b14df6d P 80f054cb213f481b90a85eb372010da4: Neither blocks nor log segments found. Creating new log.
I20260620 23:15:55.326128 30829 tablet_bootstrap.cc:492] T 32e965bf154d4035b26a1ec02b14df6d P 196905551f77460cb894f77322217edb: Bootstrap starting.
I20260620 23:15:55.326897 30828 tablet_bootstrap.cc:492] T 32e965bf154d4035b26a1ec02b14df6d P 4d35210098d240a4ac37dfcb735c820e: Bootstrap starting.
I20260620 23:15:55.328229 30827 log.cc:824] T 32e965bf154d4035b26a1ec02b14df6d P 80f054cb213f481b90a85eb372010da4: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:55.330420 30828 tablet_bootstrap.cc:654] T 32e965bf154d4035b26a1ec02b14df6d P 4d35210098d240a4ac37dfcb735c820e: Neither blocks nor log segments found. Creating new log.
I20260620 23:15:55.330416 30829 tablet_bootstrap.cc:654] T 32e965bf154d4035b26a1ec02b14df6d P 196905551f77460cb894f77322217edb: Neither blocks nor log segments found. Creating new log.
I20260620 23:15:55.330749 30827 tablet_bootstrap.cc:492] T 32e965bf154d4035b26a1ec02b14df6d P 80f054cb213f481b90a85eb372010da4: No bootstrap required, opened a new log
I20260620 23:15:55.331025 30827 ts_tablet_manager.cc:1397] T 32e965bf154d4035b26a1ec02b14df6d P 80f054cb213f481b90a85eb372010da4: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 23:15:55.331662 30828 log.cc:824] T 32e965bf154d4035b26a1ec02b14df6d P 4d35210098d240a4ac37dfcb735c820e: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:55.331662 30829 log.cc:824] T 32e965bf154d4035b26a1ec02b14df6d P 196905551f77460cb894f77322217edb: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:55.338306 30829 tablet_bootstrap.cc:492] T 32e965bf154d4035b26a1ec02b14df6d P 196905551f77460cb894f77322217edb: No bootstrap required, opened a new log
I20260620 23:15:55.338575 30829 ts_tablet_manager.cc:1397] T 32e965bf154d4035b26a1ec02b14df6d P 196905551f77460cb894f77322217edb: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260620 23:15:55.338896 30828 tablet_bootstrap.cc:492] T 32e965bf154d4035b26a1ec02b14df6d P 4d35210098d240a4ac37dfcb735c820e: No bootstrap required, opened a new log
I20260620 23:15:55.339180 30828 ts_tablet_manager.cc:1397] T 32e965bf154d4035b26a1ec02b14df6d P 4d35210098d240a4ac37dfcb735c820e: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.009s
I20260620 23:15:55.339800 30827 raft_consensus.cc:348] T 32e965bf154d4035b26a1ec02b14df6d P 80f054cb213f481b90a85eb372010da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196905551f77460cb894f77322217edb" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 46709 } } peers { permanent_uuid: "4d35210098d240a4ac37dfcb735c820e" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 36035 } } peers { permanent_uuid: "80f054cb213f481b90a85eb372010da4" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 37015 } }
I20260620 23:15:55.340217 30827 raft_consensus.cc:374] T 32e965bf154d4035b26a1ec02b14df6d P 80f054cb213f481b90a85eb372010da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:15:55.340317 30827 raft_consensus.cc:729] T 32e965bf154d4035b26a1ec02b14df6d P 80f054cb213f481b90a85eb372010da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80f054cb213f481b90a85eb372010da4, State: Initialized, Role: FOLLOWER
I20260620 23:15:55.340929 30827 consensus_queue.cc:260] T 32e965bf154d4035b26a1ec02b14df6d P 80f054cb213f481b90a85eb372010da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196905551f77460cb894f77322217edb" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 46709 } } peers { permanent_uuid: "4d35210098d240a4ac37dfcb735c820e" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 36035 } } peers { permanent_uuid: "80f054cb213f481b90a85eb372010da4" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 37015 } }
I20260620 23:15:55.342151 30809 heartbeater.cc:503] Master 127.10.191.254:37527 was elected leader, sending a full tablet report...
I20260620 23:15:55.342783 30827 ts_tablet_manager.cc:1428] T 32e965bf154d4035b26a1ec02b14df6d P 80f054cb213f481b90a85eb372010da4: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260620 23:15:55.347528 30829 raft_consensus.cc:348] T 32e965bf154d4035b26a1ec02b14df6d P 196905551f77460cb894f77322217edb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196905551f77460cb894f77322217edb" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 46709 } } peers { permanent_uuid: "4d35210098d240a4ac37dfcb735c820e" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 36035 } } peers { permanent_uuid: "80f054cb213f481b90a85eb372010da4" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 37015 } }
I20260620 23:15:55.347971 30829 raft_consensus.cc:374] T 32e965bf154d4035b26a1ec02b14df6d P 196905551f77460cb894f77322217edb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:15:55.348078 30829 raft_consensus.cc:729] T 32e965bf154d4035b26a1ec02b14df6d P 196905551f77460cb894f77322217edb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 196905551f77460cb894f77322217edb, State: Initialized, Role: FOLLOWER
I20260620 23:15:55.348696 30829 consensus_queue.cc:260] T 32e965bf154d4035b26a1ec02b14df6d P 196905551f77460cb894f77322217edb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196905551f77460cb894f77322217edb" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 46709 } } peers { permanent_uuid: "4d35210098d240a4ac37dfcb735c820e" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 36035 } } peers { permanent_uuid: "80f054cb213f481b90a85eb372010da4" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 37015 } }
I20260620 23:15:55.348752 30828 raft_consensus.cc:348] T 32e965bf154d4035b26a1ec02b14df6d P 4d35210098d240a4ac37dfcb735c820e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196905551f77460cb894f77322217edb" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 46709 } } peers { permanent_uuid: "4d35210098d240a4ac37dfcb735c820e" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 36035 } } peers { permanent_uuid: "80f054cb213f481b90a85eb372010da4" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 37015 } }
I20260620 23:15:55.349126 30828 raft_consensus.cc:374] T 32e965bf154d4035b26a1ec02b14df6d P 4d35210098d240a4ac37dfcb735c820e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:15:55.349236 30828 raft_consensus.cc:729] T 32e965bf154d4035b26a1ec02b14df6d P 4d35210098d240a4ac37dfcb735c820e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d35210098d240a4ac37dfcb735c820e, State: Initialized, Role: FOLLOWER
I20260620 23:15:55.349782 30828 consensus_queue.cc:260] T 32e965bf154d4035b26a1ec02b14df6d P 4d35210098d240a4ac37dfcb735c820e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196905551f77460cb894f77322217edb" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 46709 } } peers { permanent_uuid: "4d35210098d240a4ac37dfcb735c820e" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 36035 } } peers { permanent_uuid: "80f054cb213f481b90a85eb372010da4" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 37015 } }
I20260620 23:15:55.350889 30676 heartbeater.cc:503] Master 127.10.191.254:37527 was elected leader, sending a full tablet report...
I20260620 23:15:55.351367 30828 ts_tablet_manager.cc:1428] T 32e965bf154d4035b26a1ec02b14df6d P 4d35210098d240a4ac37dfcb735c820e: Time spent starting tablet: real 0.012s user 0.005s sys 0.007s
I20260620 23:15:55.352133 30829 ts_tablet_manager.cc:1428] T 32e965bf154d4035b26a1ec02b14df6d P 196905551f77460cb894f77322217edb: Time spent starting tablet: real 0.013s user 0.008s sys 0.006s
W20260620 23:15:55.470647 30543 tablet.cc:2369] T 32e965bf154d4035b26a1ec02b14df6d P 196905551f77460cb894f77322217edb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:15:55.475929 30810 tablet.cc:2369] T 32e965bf154d4035b26a1ec02b14df6d P 80f054cb213f481b90a85eb372010da4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:15:55.476989 30677 tablet.cc:2369] T 32e965bf154d4035b26a1ec02b14df6d P 4d35210098d240a4ac37dfcb735c820e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:15:55.605913 30833 raft_consensus.cc:482] T 32e965bf154d4035b26a1ec02b14df6d P 80f054cb213f481b90a85eb372010da4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:15:55.606238 30833 raft_consensus.cc:504] T 32e965bf154d4035b26a1ec02b14df6d P 80f054cb213f481b90a85eb372010da4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196905551f77460cb894f77322217edb" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 46709 } } peers { permanent_uuid: "4d35210098d240a4ac37dfcb735c820e" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 36035 } } peers { permanent_uuid: "80f054cb213f481b90a85eb372010da4" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 37015 } }
I20260620 23:15:55.608013 30833 leader_election.cc:288] T 32e965bf154d4035b26a1ec02b14df6d P 80f054cb213f481b90a85eb372010da4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 196905551f77460cb894f77322217edb (127.10.191.193:46709), 4d35210098d240a4ac37dfcb735c820e (127.10.191.194:36035)
I20260620 23:15:55.612125 30834 raft_consensus.cc:482] T 32e965bf154d4035b26a1ec02b14df6d P 196905551f77460cb894f77322217edb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:15:55.612406 30834 raft_consensus.cc:504] T 32e965bf154d4035b26a1ec02b14df6d P 196905551f77460cb894f77322217edb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196905551f77460cb894f77322217edb" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 46709 } } peers { permanent_uuid: "4d35210098d240a4ac37dfcb735c820e" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 36035 } } peers { permanent_uuid: "80f054cb213f481b90a85eb372010da4" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 37015 } }
I20260620 23:15:55.614840 30811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:55.608306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:46709 (local address 127.10.191.195:32887)
0620 23:15:55.608500 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:55.608517 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:55.608659 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:55.609237 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:55.609241 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:55.609262 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:55.609428 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:55.609436 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:55.610770 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:55.610775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:55.612611 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:55.612624 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:55.614061 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:55.614069 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:55.614250 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:55.614274 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 23:15:55.614376 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 23:15:55.614510 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":21}
I20260620 23:15:55.615739 30822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:55.608528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:32887 (local address 127.10.191.193:46709)
0620 23:15:55.608762 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:55.608769 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:55.608790 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:15:55.608869 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:55.608875 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:55.608945 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:55.609034 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:55.609579 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:55.610621 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:55.612774 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:55.613915 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:55.614081 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:55.614154 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:55.615189 (+ 1035us) server_negotiation.cc:299] Negotiation successful
0620 23:15:55.615379 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 23:15:55.616238 30825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:55.608932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:33669 (local address 127.10.191.194:36035)
0620 23:15:55.609154 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:55.609161 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:55.609538 (+ 377us) server_negotiation.cc:407] Connection header received
0620 23:15:55.609708 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:55.609712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:55.609768 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:55.609834 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:55.610390 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:55.611371 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:55.613513 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:55.614734 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:55.614928 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:55.615034 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:55.615678 (+ 644us) server_negotiation.cc:299] Negotiation successful
0620 23:15:55.615859 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 23:15:55.616753 30836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:55.608776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.194:36035 (local address 127.10.191.195:33669)
0620 23:15:55.609426 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:55.609441 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:55.609579 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:55.609878 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:55.609885 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:55.609911 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:55.610245 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:55.610256 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:55.611522 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:55.611526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:55.613333 (+ 1807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:55.613347 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:55.615187 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:55.615196 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:55.615448 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:55.615497 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 23:15:55.616222 (+ 725us) client_negotiation.cc:240] Negotiation successful
0620 23:15:55.616364 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"mutex_wait_us":1,"thread_start_us":329,"threads_started":1}
I20260620 23:15:55.617009 30497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32e965bf154d4035b26a1ec02b14df6d" candidate_uuid: "80f054cb213f481b90a85eb372010da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "196905551f77460cb894f77322217edb" is_pre_election: true
I20260620 23:15:55.617584 30497 raft_consensus.cc:2454] T 32e965bf154d4035b26a1ec02b14df6d P 196905551f77460cb894f77322217edb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80f054cb213f481b90a85eb372010da4 in term 0.
I20260620 23:15:55.618281 30834 leader_election.cc:288] T 32e965bf154d4035b26a1ec02b14df6d P 196905551f77460cb894f77322217edb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d35210098d240a4ac37dfcb735c820e (127.10.191.194:36035), 80f054cb213f481b90a85eb372010da4 (127.10.191.195:37015)
I20260620 23:15:55.618474 30699 leader_election.cc:302] T 32e965bf154d4035b26a1ec02b14df6d P 80f054cb213f481b90a85eb372010da4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 196905551f77460cb894f77322217edb, 80f054cb213f481b90a85eb372010da4; no voters:
I20260620 23:15:55.618990 30833 raft_consensus.cc:2781] T 32e965bf154d4035b26a1ec02b14df6d P 80f054cb213f481b90a85eb372010da4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:15:55.619125 30833 raft_consensus.cc:482] T 32e965bf154d4035b26a1ec02b14df6d P 80f054cb213f481b90a85eb372010da4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:15:55.619246 30833 raft_consensus.cc:3037] T 32e965bf154d4035b26a1ec02b14df6d P 80f054cb213f481b90a85eb372010da4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:15:55.622608 30833 raft_consensus.cc:504] T 32e965bf154d4035b26a1ec02b14df6d P 80f054cb213f481b90a85eb372010da4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196905551f77460cb894f77322217edb" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 46709 } } peers { permanent_uuid: "4d35210098d240a4ac37dfcb735c820e" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 36035 } } peers { permanent_uuid: "80f054cb213f481b90a85eb372010da4" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 37015 } }
I20260620 23:15:55.623261 30833 leader_election.cc:288] T 32e965bf154d4035b26a1ec02b14df6d P 80f054cb213f481b90a85eb372010da4 [CANDIDATE]: Term 1 election: Requested vote from peers 196905551f77460cb894f77322217edb (127.10.191.193:46709), 4d35210098d240a4ac37dfcb735c820e (127.10.191.194:36035)
I20260620 23:15:55.624126 30497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32e965bf154d4035b26a1ec02b14df6d" candidate_uuid: "80f054cb213f481b90a85eb372010da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "196905551f77460cb894f77322217edb"
I20260620 23:15:55.624354 30497 raft_consensus.cc:3037] T 32e965bf154d4035b26a1ec02b14df6d P 196905551f77460cb894f77322217edb [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:15:55.624907 30825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:55.614539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:53127 (local address 127.10.191.194:36035)
0620 23:15:55.616522 (+ 1983us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:55.616530 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:55.617501 (+ 971us) server_negotiation.cc:407] Connection header received
0620 23:15:55.617793 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:55.617798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:55.617870 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:55.617932 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:55.618836 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:55.619977 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:55.622488 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:55.623664 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:55.623805 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:55.623875 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:55.624460 (+ 585us) server_negotiation.cc:299] Negotiation successful
0620 23:15:55.624595 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1855,"threads_started":1}
I20260620 23:15:55.625573 30838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:55.614556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.194:36035 (local address 127.10.191.193:53127)
0620 23:15:55.617390 (+ 2834us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:55.617412 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:55.617611 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:55.618152 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:55.618156 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:55.618184 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:55.618652 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:55.618665 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:55.620161 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:55.620166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:55.622305 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:55.622322 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:55.624068 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:55.624073 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:55.624254 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:55.624288 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 23:15:55.625047 (+ 759us) client_negotiation.cc:240] Negotiation successful
0620 23:15:55.625194 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2679,"thread_start_us":2467,"threads_started":1}
I20260620 23:15:55.625765 30824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:55.617944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:48231 (local address 127.10.191.195:37015)
0620 23:15:55.618170 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:55.618178 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:55.618965 (+ 787us) server_negotiation.cc:407] Connection header received
0620 23:15:55.619195 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:55.619200 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:55.619262 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:55.619313 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:55.619970 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:55.621092 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:55.623465 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:55.624670 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:55.624781 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:55.624853 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:55.625430 (+ 577us) server_negotiation.cc:299] Negotiation successful
0620 23:15:55.625552 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 23:15:55.626533 30839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:55.617615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.195:37015 (local address 127.10.191.193:48231)
0620 23:15:55.618833 (+ 1218us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:55.618884 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:55.619072 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:55.619461 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:55.619465 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:55.619486 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:55.619795 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:55.619803 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:55.621261 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:55.621266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:55.623293 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:55.623307 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:55.625056 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:55.625063 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:55.625251 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:55.625283 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 23:15:55.626103 (+ 820us) client_negotiation.cc:240] Negotiation successful
0620 23:15:55.626239 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1104,"threads_started":1}
I20260620 23:15:55.626819 30631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32e965bf154d4035b26a1ec02b14df6d" candidate_uuid: "196905551f77460cb894f77322217edb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d35210098d240a4ac37dfcb735c820e" is_pre_election: true
I20260620 23:15:55.627151 30630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32e965bf154d4035b26a1ec02b14df6d" candidate_uuid: "80f054cb213f481b90a85eb372010da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d35210098d240a4ac37dfcb735c820e" is_pre_election: true
I20260620 23:15:55.627296 30629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32e965bf154d4035b26a1ec02b14df6d" candidate_uuid: "80f054cb213f481b90a85eb372010da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d35210098d240a4ac37dfcb735c820e"
I20260620 23:15:55.627547 30631 raft_consensus.cc:2454] T 32e965bf154d4035b26a1ec02b14df6d P 4d35210098d240a4ac37dfcb735c820e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 196905551f77460cb894f77322217edb in term 0.
I20260620 23:15:55.627754 30764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32e965bf154d4035b26a1ec02b14df6d" candidate_uuid: "196905551f77460cb894f77322217edb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80f054cb213f481b90a85eb372010da4" is_pre_election: true
I20260620 23:15:55.628208 30764 raft_consensus.cc:2379] T 32e965bf154d4035b26a1ec02b14df6d P 80f054cb213f481b90a85eb372010da4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 196905551f77460cb894f77322217edb in current term 1: Already voted for candidate 80f054cb213f481b90a85eb372010da4 in this term.
I20260620 23:15:55.628340 30432 leader_election.cc:302] T 32e965bf154d4035b26a1ec02b14df6d P 196905551f77460cb894f77322217edb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 196905551f77460cb894f77322217edb, 4d35210098d240a4ac37dfcb735c820e; no voters:
I20260620 23:15:55.629590 30497 raft_consensus.cc:2454] T 32e965bf154d4035b26a1ec02b14df6d P 196905551f77460cb894f77322217edb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80f054cb213f481b90a85eb372010da4 in term 1.
I20260620 23:15:55.629789 30834 raft_consensus.cc:2741] T 32e965bf154d4035b26a1ec02b14df6d P 196905551f77460cb894f77322217edb [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260620 23:15:55.630060 30699 leader_election.cc:302] T 32e965bf154d4035b26a1ec02b14df6d P 80f054cb213f481b90a85eb372010da4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 196905551f77460cb894f77322217edb, 80f054cb213f481b90a85eb372010da4; no voters: 4d35210098d240a4ac37dfcb735c820e
I20260620 23:15:55.630376 30833 raft_consensus.cc:2781] T 32e965bf154d4035b26a1ec02b14df6d P 80f054cb213f481b90a85eb372010da4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:15:55.630909 30833 raft_consensus.cc:686] T 32e965bf154d4035b26a1ec02b14df6d P 80f054cb213f481b90a85eb372010da4 [term 1 LEADER]: Becoming Leader. State: Replica: 80f054cb213f481b90a85eb372010da4, State: Running, Role: LEADER
I20260620 23:15:55.631398 30833 consensus_queue.cc:237] T 32e965bf154d4035b26a1ec02b14df6d P 80f054cb213f481b90a85eb372010da4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196905551f77460cb894f77322217edb" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 46709 } } peers { permanent_uuid: "4d35210098d240a4ac37dfcb735c820e" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 36035 } } peers { permanent_uuid: "80f054cb213f481b90a85eb372010da4" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 37015 } }
I20260620 23:15:55.635598 30353 catalog_manager.cc:5495] T 32e965bf154d4035b26a1ec02b14df6d P 80f054cb213f481b90a85eb372010da4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80f054cb213f481b90a85eb372010da4 (127.10.191.195). New cstate: current_term: 1 leader_uuid: "80f054cb213f481b90a85eb372010da4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196905551f77460cb894f77322217edb" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 46709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d35210098d240a4ac37dfcb735c820e" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 36035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80f054cb213f481b90a85eb372010da4" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 37015 } health_report { overall_health: HEALTHY } } }
I20260620 23:15:55.671440 30824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:55.666222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51514 (local address 127.10.191.195:37015)
0620 23:15:55.666431 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:55.666439 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:55.666566 (+ 127us) server_negotiation.cc:407] Connection header received
0620 23:15:55.666723 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:55.666728 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:55.666812 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:55.666996 (+ 184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:15:55.667315 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:55.668029 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:55.669456 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:55.669910 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:55.670001 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:55.670414 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:15:55.670959 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:15:55.671066 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:55.671145 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 23:15:55.671208 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 23:15:55.678828 30630 raft_consensus.cc:3037] T 32e965bf154d4035b26a1ec02b14df6d P 4d35210098d240a4ac37dfcb735c820e [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:15:55.679037 30497 raft_consensus.cc:1261] T 32e965bf154d4035b26a1ec02b14df6d P 196905551f77460cb894f77322217edb [term 1 FOLLOWER]: Refusing update from remote peer 80f054cb213f481b90a85eb372010da4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:15:55.679875 30840 consensus_queue.cc:1035] T 32e965bf154d4035b26a1ec02b14df6d P 80f054cb213f481b90a85eb372010da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "196905551f77460cb894f77322217edb" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 46709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:15:55.681639 30630 raft_consensus.cc:1261] T 32e965bf154d4035b26a1ec02b14df6d P 4d35210098d240a4ac37dfcb735c820e [term 1 FOLLOWER]: Refusing update from remote peer 80f054cb213f481b90a85eb372010da4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:15:55.682361 30840 consensus_queue.cc:1035] T 32e965bf154d4035b26a1ec02b14df6d P 80f054cb213f481b90a85eb372010da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d35210098d240a4ac37dfcb735c820e" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 36035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:15:55.691184 30844 mvcc.cc:205] Tried to move back new op lower bound from 7299061168851951616 to 7299061168669700096. Current Snapshot: MvccSnapshot[applied={T|T < 7299061168851951616}]
I20260620 23:15:55.704375 30353 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:15:55.704653 30353 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:15:55.708261 30353 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 82436260b6e84e0a8959d79b5aa4dde2: Sending DeleteTablet for 3 replicas of tablet 32e965bf154d4035b26a1ec02b14df6d
I20260620 23:15:55.709414 30744 tablet_service.cc:1514] Processing DeleteTablet for tablet 32e965bf154d4035b26a1ec02b14df6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:15:55 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:51508
I20260620 23:15:55.709672 30477 tablet_service.cc:1514] Processing DeleteTablet for tablet 32e965bf154d4035b26a1ec02b14df6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:15:55 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:34224
I20260620 23:15:55.709678 30610 tablet_service.cc:1514] Processing DeleteTablet for tablet 32e965bf154d4035b26a1ec02b14df6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:15:55 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:45646
I20260620 23:15:55.710215 30611 tablet_service.cc:1514] Processing DeleteTablet for tablet 32e965bf154d4035b26a1ec02b14df6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:15:55 UTC) from {username='kudu', principal='kudu/127.10.191.254@KRBTEST.COM'} at 127.0.0.1:45646
I20260620 23:15:55.712580 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 30414
I20260620 23:15:55.739944 30853 tablet_replica.cc:330] T 32e965bf154d4035b26a1ec02b14df6d P 196905551f77460cb894f77322217edb: stopping tablet replica
I20260620 23:15:55.742409 30853 raft_consensus.cc:2229] T 32e965bf154d4035b26a1ec02b14df6d P 196905551f77460cb894f77322217edb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:15:55.744624 30853 raft_consensus.cc:2258] T 32e965bf154d4035b26a1ec02b14df6d P 196905551f77460cb894f77322217edb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:15:55.749111 30853 ts_tablet_manager.cc:1905] T 32e965bf154d4035b26a1ec02b14df6d P 196905551f77460cb894f77322217edb: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 23:15:55 dist-test-slave-jkbq krb5kdc[30308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.193@KRBTEST.COM
I20260620 23:15:55.759251 30852 tablet_replica.cc:330] T 32e965bf154d4035b26a1ec02b14df6d P 80f054cb213f481b90a85eb372010da4: stopping tablet replica
I20260620 23:15:55.759793 30852 raft_consensus.cc:2229] T 32e965bf154d4035b26a1ec02b14df6d P 80f054cb213f481b90a85eb372010da4 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:15:55.760551 30852 raft_consensus.cc:2258] T 32e965bf154d4035b26a1ec02b14df6d P 80f054cb213f481b90a85eb372010da4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:15:55.761349 30854 tablet_replica.cc:330] T 32e965bf154d4035b26a1ec02b14df6d P 4d35210098d240a4ac37dfcb735c820e: stopping tablet replica
I20260620 23:15:55.761778 30854 raft_consensus.cc:2229] T 32e965bf154d4035b26a1ec02b14df6d P 4d35210098d240a4ac37dfcb735c820e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:15:55.762331 30854 raft_consensus.cc:2258] T 32e965bf154d4035b26a1ec02b14df6d P 4d35210098d240a4ac37dfcb735c820e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:15:55.763571 30852 ts_tablet_manager.cc:1905] T 32e965bf154d4035b26a1ec02b14df6d P 80f054cb213f481b90a85eb372010da4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:15:55.764338 30854 ts_tablet_manager.cc:1905] T 32e965bf154d4035b26a1ec02b14df6d P 4d35210098d240a4ac37dfcb735c820e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:15:55.768749 30822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:55.714095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34228 (local address 127.10.191.193:46709)
0620 23:15:55.714319 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:55.714328 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:55.739069 (+ 24741us) server_negotiation.cc:407] Connection header received
0620 23:15:55.740482 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:55.740493 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:55.740606 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:55.740716 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:55.745149 (+ 4433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:55.746601 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:55.751846 (+ 5245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:55.752844 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:55.752977 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:55.760429 (+ 7452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:55.760449 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:55.760473 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:55.760554 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:55.764859 (+ 4305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:55.765897 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:55.765906 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:55.765909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:55.766067 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:55.766608 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:55.766617 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:55.766623 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:55.767082 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:55.767363 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:55.768017 (+ 654us) server_negotiation.cc:299] Negotiation successful
0620 23:15:55.768264 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 23:15:55.770484 30537 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:15:55.771255 30854 ts_tablet_manager.cc:1918] T 32e965bf154d4035b26a1ec02b14df6d P 4d35210098d240a4ac37dfcb735c820e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:15:55.771410 30854 log.cc:1196] T 32e965bf154d4035b26a1ec02b14df6d P 4d35210098d240a4ac37dfcb735c820e: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-1/wal/wals/32e965bf154d4035b26a1ec02b14df6d
I20260620 23:15:55.771533 30852 ts_tablet_manager.cc:1918] T 32e965bf154d4035b26a1ec02b14df6d P 80f054cb213f481b90a85eb372010da4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:15:55.771723 30852 log.cc:1196] T 32e965bf154d4035b26a1ec02b14df6d P 80f054cb213f481b90a85eb372010da4: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-2/wal/wals/32e965bf154d4035b26a1ec02b14df6d
I20260620 23:15:55.771924 30854 ts_tablet_manager.cc:1939] T 32e965bf154d4035b26a1ec02b14df6d P 4d35210098d240a4ac37dfcb735c820e: Deleting consensus metadata
I20260620 23:15:55.772367 30852 ts_tablet_manager.cc:1939] T 32e965bf154d4035b26a1ec02b14df6d P 80f054cb213f481b90a85eb372010da4: Deleting consensus metadata
I20260620 23:15:55.774470 30338 catalog_manager.cc:4841] TS 80f054cb213f481b90a85eb372010da4 (127.10.191.195:37015): tablet 32e965bf154d4035b26a1ec02b14df6d (table test-table [id=96b5c162c326437f860b012d9fd78aad]) successfully deleted
I20260620 23:15:55.775552 30339 catalog_manager.cc:4841] TS 4d35210098d240a4ac37dfcb735c820e (127.10.191.194:36035): tablet 32e965bf154d4035b26a1ec02b14df6d (table test-table [id=96b5c162c326437f860b012d9fd78aad]) successfully deleted
W20260620 23:15:55.776954 30339 catalog_manager.cc:4808] TS 4d35210098d240a4ac37dfcb735c820e (127.10.191.194:36035): delete failed for tablet 32e965bf154d4035b26a1ec02b14df6d because the tablet was not found. No further retry: Not found: Tablet not found: 32e965bf154d4035b26a1ec02b14df6d
I20260620 23:15:55.867017 30853 ts_tablet_manager.cc:1918] T 32e965bf154d4035b26a1ec02b14df6d P 196905551f77460cb894f77322217edb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:15:55.867300 30853 log.cc:1196] T 32e965bf154d4035b26a1ec02b14df6d P 196905551f77460cb894f77322217edb: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781997042893763-11007-0/minicluster-data/ts-0/wal/wals/32e965bf154d4035b26a1ec02b14df6d
I20260620 23:15:55.867637 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 30414
I20260620 23:15:55.885717 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 30548
W20260620 23:15:55.886015 30339 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.191.193:46709 (error 108)
I20260620 23:15:55.888063 30858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:55.887265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:46709 (local address 127.0.0.1:34232)
0620 23:15:55.887723 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:55.887855 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.193:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":221,"threads_started":1}
W20260620 23:15:55.888353 30339 catalog_manager.cc:4584] TS 196905551f77460cb894f77322217edb (127.10.191.193:46709): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 32e965bf154d4035b26a1ec02b14df6d: Network error: Client connection negotiation failed: client connection to 127.10.191.193:46709: connect: Connection refused (error 111)
Jun 20 23:15:55 dist-test-slave-jkbq krb5kdc[30308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.194@KRBTEST.COM
I20260620 23:15:55.902951 30825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:55.886639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45652 (local address 127.10.191.194:36035)
0620 23:15:55.886831 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:55.886837 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:55.886927 (+ 90us) server_negotiation.cc:407] Connection header received
0620 23:15:55.887203 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:55.887209 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:55.887329 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:55.887452 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:55.889012 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:55.890091 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:55.891682 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:55.892333 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:55.892461 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:55.896581 (+ 4120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:55.896599 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:55.896619 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:55.896662 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:55.900290 (+ 3628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:55.900920 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:55.900930 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:55.900934 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:55.901091 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:55.901429 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:55.901435 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:55.901439 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:55.901849 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:55.902165 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:55.902323 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 23:15:55.902559 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 23:15:55.903692 30671 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:15:55.941980 30858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:55.941456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:46709 (local address 127.0.0.1:34240)
0620 23:15:55.941693 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:55.941801 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.193:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 23:15:55.943315 30858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:55.942936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:46709 (local address 127.0.0.1:34246)
0620 23:15:55.943089 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:55.943170 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.193:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 23:15:55.993999 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 30548
I20260620 23:15:56.003890 30858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:56.003452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:46709 (local address 127.0.0.1:34262)
0620 23:15:56.003650 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:56.003739 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.193:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 23:15:56.004941 30858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:56.004584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:46709 (local address 127.0.0.1:34274)
0620 23:15:56.004727 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:56.004799 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.193:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 23:15:56.017668 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 30681
Jun 20 23:15:56 dist-test-slave-jkbq krb5kdc[30308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.195@KRBTEST.COM
I20260620 23:15:56.036302 30824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:56.018502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51528 (local address 127.10.191.195:37015)
0620 23:15:56.018682 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:56.018689 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:56.018706 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:15:56.018944 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:56.018948 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:56.019023 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:56.019106 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:56.020829 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:56.021572 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:56.023463 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:56.023898 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:56.024003 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:56.029277 (+ 5274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:56.029294 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:56.029316 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:15:56.029370 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:56.033490 (+ 4120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:56.034098 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:56.034109 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:56.034112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:56.034272 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:15:56.034614 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:15:56.034619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:15:56.034623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:15:56.035177 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:56.035492 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:56.035672 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 23:15:56.035938 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 23:15:56.037024 30804 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:15:56.086109 30858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:56.085587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:46709 (local address 127.0.0.1:34282)
0620 23:15:56.085811 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:56.085924 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.193:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 23:15:56.087307 30858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:56.086814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:46709 (local address 127.0.0.1:34290)
0620 23:15:56.087090 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:56.087170 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.193:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
I20260620 23:15:56.159444 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 30681
I20260620 23:15:56.179010 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 30321
I20260620 23:15:56.180106 30383 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:15:56.292609 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 30321
I20260620 23:15:56.292706 30858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:56.291911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:46709 (local address 127.0.0.1:34302)
0620 23:15:56.292428 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:56.292541 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.193:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363}
2026-06-20T23:15:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4448 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T23:15:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:15:56Z Disabled control of system clock
I20260620 23:15:56.367396 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:44105
--webserver_interface=127.10.191.254
--webserver_port=0
--builtin_ntp_servers=127.10.191.212:44161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:44105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:15:56.693054 30865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:56.693336 30865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:56.693423 30865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:56.701303 30865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:15:56.701406 30865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:56.701443 30865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:56.701500 30865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:15:56.701543 30865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:15:56.710593 30865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:44161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:44105
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:44105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:15:56.711861 30865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:56.713546 30865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:56.725119 30870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:56.725242 30871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:56.725406 30873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:56.726137 30865 server_base.cc:1034] running on GCE node
I20260620 23:15:56.726972 30865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:56.728403 30865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:56.729619 30865 hybrid_clock.cc:648] HybridClock initialized: now 1781997356729563 us; error 47 us; skew 500 ppm
I20260620 23:15:56.729959 30865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:15:56.732213 30865 webserver.cc:460] Webserver started at http://127.10.191.254:34937/ using document root <none> and password file <none>
I20260620 23:15:56.732882 30865 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:56.733011 30865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:56.733322 30865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:15:56.736161 30865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "7e537d5211dc469c9ffce23243b6c1ee"
format_stamp: "Formatted at 2026-06-20 23:15:56 on dist-test-slave-jkbq"
server_key: "6fadfce450bb0af123f8f2305bdaffef"
server_key_iv: "254e60936cd643f998cbeace77f2f726"
server_key_version: "encryptionkey@0"
I20260620 23:15:56.736953 30865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "7e537d5211dc469c9ffce23243b6c1ee"
format_stamp: "Formatted at 2026-06-20 23:15:56 on dist-test-slave-jkbq"
server_key: "6fadfce450bb0af123f8f2305bdaffef"
server_key_iv: "254e60936cd643f998cbeace77f2f726"
server_key_version: "encryptionkey@0"
I20260620 23:15:56.741935 30865 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 23:15:56.745460 30879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:56.746825 30865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:15:56.747022 30865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "7e537d5211dc469c9ffce23243b6c1ee"
format_stamp: "Formatted at 2026-06-20 23:15:56 on dist-test-slave-jkbq"
server_key: "6fadfce450bb0af123f8f2305bdaffef"
server_key_iv: "254e60936cd643f998cbeace77f2f726"
server_key_version: "encryptionkey@0"
I20260620 23:15:56.747236 30865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:56.773715 30865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:56.774600 30865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:56.774955 30865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:56.793371 30865 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:44105
I20260620 23:15:56.793401 30930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:44105 every 8 connection(s)
I20260620 23:15:56.794670 30865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:15:56.798269 30931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:15:56.799502 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 30865
I20260620 23:15:56.799690 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:15:56.800189 11007 external_mini_cluster.cc:1428] Setting key 4587d6ce7a9120db09d2d81a71f0d5c5
I20260620 23:15:56.812747 30931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e537d5211dc469c9ffce23243b6c1ee: Bootstrap starting.
I20260620 23:15:56.814852 30934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:56.802096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34210 (local address 127.10.191.254:44105)
0620 23:15:56.803260 (+ 1164us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:56.803289 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:56.803339 (+ 50us) server_negotiation.cc:407] Connection header received
0620 23:15:56.804967 (+ 1628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:56.804994 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:56.805591 (+ 597us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:56.806018 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:56.806551 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:56.808414 (+ 1863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:56.809806 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:56.810858 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:56.811057 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:56.812120 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:56.812131 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:56.812148 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:15:56.812842 (+ 694us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:56.812942 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:15:56.812966 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:56.813069 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:56.813344 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 23:15:56.813631 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":340,"threads_started":1}
I20260620 23:15:56.816325 30931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e537d5211dc469c9ffce23243b6c1ee: Neither blocks nor log segments found. Creating new log.
I20260620 23:15:56.817359 30931 log.cc:824] T 00000000000000000000000000000000 P 7e537d5211dc469c9ffce23243b6c1ee: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:56.819887 30931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e537d5211dc469c9ffce23243b6c1ee: No bootstrap required, opened a new log
I20260620 23:15:56.826656 30931 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e537d5211dc469c9ffce23243b6c1ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e537d5211dc469c9ffce23243b6c1ee" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 44105 } }
I20260620 23:15:56.827085 30931 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e537d5211dc469c9ffce23243b6c1ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:15:56.827180 30931 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e537d5211dc469c9ffce23243b6c1ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e537d5211dc469c9ffce23243b6c1ee, State: Initialized, Role: FOLLOWER
I20260620 23:15:56.827709 30931 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e537d5211dc469c9ffce23243b6c1ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e537d5211dc469c9ffce23243b6c1ee" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 44105 } }
I20260620 23:15:56.827951 30931 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e537d5211dc469c9ffce23243b6c1ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:15:56.828079 30931 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e537d5211dc469c9ffce23243b6c1ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:15:56.828243 30931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e537d5211dc469c9ffce23243b6c1ee [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:15:56.830315 30931 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e537d5211dc469c9ffce23243b6c1ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e537d5211dc469c9ffce23243b6c1ee" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 44105 } }
I20260620 23:15:56.830796 30931 leader_election.cc:302] T 00000000000000000000000000000000 P 7e537d5211dc469c9ffce23243b6c1ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e537d5211dc469c9ffce23243b6c1ee; no voters:
I20260620 23:15:56.831429 30931 leader_election.cc:288] T 00000000000000000000000000000000 P 7e537d5211dc469c9ffce23243b6c1ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:15:56.831614 30936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e537d5211dc469c9ffce23243b6c1ee [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:15:56.832325 30936 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e537d5211dc469c9ffce23243b6c1ee [term 1 LEADER]: Becoming Leader. State: Replica: 7e537d5211dc469c9ffce23243b6c1ee, State: Running, Role: LEADER
I20260620 23:15:56.832823 30936 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e537d5211dc469c9ffce23243b6c1ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e537d5211dc469c9ffce23243b6c1ee" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 44105 } }
I20260620 23:15:56.833452 30931 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e537d5211dc469c9ffce23243b6c1ee [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:15:56.835338 30937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e537d5211dc469c9ffce23243b6c1ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e537d5211dc469c9ffce23243b6c1ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e537d5211dc469c9ffce23243b6c1ee" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 44105 } } }
I20260620 23:15:56.835743 30937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e537d5211dc469c9ffce23243b6c1ee [sys.catalog]: This master's current role is: LEADER
I20260620 23:15:56.835637 30938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e537d5211dc469c9ffce23243b6c1ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e537d5211dc469c9ffce23243b6c1ee. Latest consensus state: current_term: 1 leader_uuid: "7e537d5211dc469c9ffce23243b6c1ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e537d5211dc469c9ffce23243b6c1ee" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 44105 } } }
I20260620 23:15:56.836169 30938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e537d5211dc469c9ffce23243b6c1ee [sys.catalog]: This master's current role is: LEADER
I20260620 23:15:56.838171 30941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:15:56.843775 30941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:15:56.852752 30941 catalog_manager.cc:1347] Generated new cluster ID: 25cd0587fb114afb9f00bc3289583682
I20260620 23:15:56.852880 30941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:15:56.862303 30941 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:15:56.863615 30941 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:15:56.873770 30941 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e537d5211dc469c9ffce23243b6c1ee: Generated new TSK 0
I20260620 23:15:56.874605 30941 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:15:56.885020 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.193:0
--local_ip_for_outbound_sockets=127.10.191.193
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:44105
--builtin_ntp_servers=127.10.191.212:44161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:15:57.214668 30955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:57.215019 30955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:57.215116 30955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:57.222177 30955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:57.222265 30955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:57.222365 30955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.193
I20260620 23:15:57.231113 30955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:44161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.191.193
--webserver_port=0
--tserver_master_addrs=127.10.191.254:44105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.193
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:15:57.232308 30955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:57.234128 30955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:57.247340 30961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:57.247402 30960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:57.247408 30963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:57.248008 30955 server_base.cc:1034] running on GCE node
I20260620 23:15:57.248610 30955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:57.249483 30955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:57.250710 30955 hybrid_clock.cc:648] HybridClock initialized: now 1781997357250666 us; error 53 us; skew 500 ppm
I20260620 23:15:57.251096 30955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:15:57.253217 30955 webserver.cc:460] Webserver started at http://127.10.191.193:45905/ using document root <none> and password file <none>
I20260620 23:15:57.253886 30955 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:57.254010 30955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:57.254312 30955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:15:57.257153 30955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-0/data/instance:
uuid: "0a407a37f91344899ee365cdaae9e562"
format_stamp: "Formatted at 2026-06-20 23:15:57 on dist-test-slave-jkbq"
server_key: "4ce9b3efec35f0181db560cece1efd00"
server_key_iv: "32369f12342c9cfb351ca560438b772e"
server_key_version: "encryptionkey@0"
I20260620 23:15:57.257921 30955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance:
uuid: "0a407a37f91344899ee365cdaae9e562"
format_stamp: "Formatted at 2026-06-20 23:15:57 on dist-test-slave-jkbq"
server_key: "4ce9b3efec35f0181db560cece1efd00"
server_key_iv: "32369f12342c9cfb351ca560438b772e"
server_key_version: "encryptionkey@0"
I20260620 23:15:57.263027 30955 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 23:15:57.266345 30969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:57.267593 30955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:15:57.267765 30955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-0/wal
uuid: "0a407a37f91344899ee365cdaae9e562"
format_stamp: "Formatted at 2026-06-20 23:15:57 on dist-test-slave-jkbq"
server_key: "4ce9b3efec35f0181db560cece1efd00"
server_key_iv: "32369f12342c9cfb351ca560438b772e"
server_key_version: "encryptionkey@0"
I20260620 23:15:57.267985 30955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:57.283360 30955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:57.284129 30955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:57.284382 30955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:57.285216 30955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:15:57.286516 30955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:15:57.286602 30955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:57.286674 30955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:15:57.286727 30955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:57.319285 30955 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.193:42789
I20260620 23:15:57.319334 31081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.193:42789 every 8 connection(s)
I20260620 23:15:57.320566 30955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-0/data/info.pb
I20260620 23:15:57.328732 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 30955
I20260620 23:15:57.328895 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-0/wal/instance
I20260620 23:15:57.329314 11007 external_mini_cluster.cc:1428] Setting key 66c399c5c61fda32379f4ae4e434d72a
I20260620 23:15:57.331974 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.194:0
--local_ip_for_outbound_sockets=127.10.191.194
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:44105
--builtin_ntp_servers=127.10.191.212:44161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 23:15:57.333063 31084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:57.322994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:46559 (local address 127.10.191.254:44105)
0620 23:15:57.323422 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:57.323426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:57.324505 (+ 1079us) server_negotiation.cc:407] Connection header received
0620 23:15:57.326272 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:57.326277 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:57.326365 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:57.326495 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:57.327613 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:57.328341 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:57.330169 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:57.330635 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:57.330741 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:57.332203 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:57.332209 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:57.332213 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:15:57.332233 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:57.332293 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:15:57.332307 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:57.332404 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:57.332626 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 23:15:57.332716 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":228,"threads_started":1}
I20260620 23:15:57.334077 31085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:57.323501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:44105 (local address 127.10.191.193:46559)
0620 23:15:57.324348 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:57.324383 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:57.325999 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:57.326727 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:57.326734 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:57.326820 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:57.327446 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:57.327462 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:57.328525 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:57.328529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:57.330045 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:57.330057 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:57.330888 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:57.330896 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:57.331182 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:57.331907 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:15:57.331910 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:57.331944 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:15:57.332013 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:57.332403 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:57.332417 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:57.332426 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 23:15:57.332600 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 23:15:57.332954 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":250,"threads_started":1}
I20260620 23:15:57.335659 31082 heartbeater.cc:347] Connected to a master server at 127.10.191.254:44105
I20260620 23:15:57.335938 31082 heartbeater.cc:464] Registering TS with master...
I20260620 23:15:57.336670 31082 heartbeater.cc:511] Master 127.10.191.254:44105 requested a full tablet report, sending...
I20260620 23:15:57.339057 30896 ts_manager.cc:194] Registered new tserver with Master: 0a407a37f91344899ee365cdaae9e562 (127.10.191.193:42789)
I20260620 23:15:57.340782 30896 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.191.193:46559
W20260620 23:15:57.653329 31086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:57.653607 31086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:57.653731 31086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:57.660984 31086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:57.661085 31086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:57.661247 31086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.194
I20260620 23:15:57.670316 31086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:44161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.191.194
--webserver_port=0
--tserver_master_addrs=127.10.191.254:44105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.194
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:15:57.671624 31086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:57.673386 31086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:57.686546 31091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:57.686617 31094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:57.686630 31092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:57.692376 31086 server_base.cc:1034] running on GCE node
I20260620 23:15:57.692983 31086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:57.693919 31086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:57.695143 31086 hybrid_clock.cc:648] HybridClock initialized: now 1781997357695146 us; error 112 us; skew 500 ppm
I20260620 23:15:57.695461 31086 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:15:57.697712 31086 webserver.cc:460] Webserver started at http://127.10.191.194:44779/ using document root <none> and password file <none>
I20260620 23:15:57.698397 31086 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:57.698529 31086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:57.698834 31086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:15:57.701666 31086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-1/data/instance:
uuid: "5b0cccf352be4a99872ad069cd705457"
format_stamp: "Formatted at 2026-06-20 23:15:57 on dist-test-slave-jkbq"
server_key: "753042a33cb9273b03b0edbbb7d4381f"
server_key_iv: "d4e4e56a90f315861f286cb6c8255f47"
server_key_version: "encryptionkey@0"
I20260620 23:15:57.702461 31086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance:
uuid: "5b0cccf352be4a99872ad069cd705457"
format_stamp: "Formatted at 2026-06-20 23:15:57 on dist-test-slave-jkbq"
server_key: "753042a33cb9273b03b0edbbb7d4381f"
server_key_iv: "d4e4e56a90f315861f286cb6c8255f47"
server_key_version: "encryptionkey@0"
I20260620 23:15:57.707573 31086 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260620 23:15:57.710842 31100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:57.712150 31086 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:15:57.712323 31086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-1/wal
uuid: "5b0cccf352be4a99872ad069cd705457"
format_stamp: "Formatted at 2026-06-20 23:15:57 on dist-test-slave-jkbq"
server_key: "753042a33cb9273b03b0edbbb7d4381f"
server_key_iv: "d4e4e56a90f315861f286cb6c8255f47"
server_key_version: "encryptionkey@0"
I20260620 23:15:57.712553 31086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:57.750202 31086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:57.751152 31086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:57.751462 31086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:57.752498 31086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:15:57.753942 31086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:15:57.754062 31086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:57.754199 31086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:15:57.754287 31086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:57.787117 31086 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.194:42457
I20260620 23:15:57.787202 31212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.194:42457 every 8 connection(s)
I20260620 23:15:57.788422 31086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-1/data/info.pb
I20260620 23:15:57.798400 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 31086
I20260620 23:15:57.798568 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-1/wal/instance
I20260620 23:15:57.799124 11007 external_mini_cluster.cc:1428] Setting key 5f1a688916930d11299ac7919dfe1235
I20260620 23:15:57.801422 31084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:57.790991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.194:50921 (local address 127.10.191.254:44105)
0620 23:15:57.791167 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:57.791172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:57.792575 (+ 1403us) server_negotiation.cc:407] Connection header received
0620 23:15:57.794338 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:57.794344 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:57.794436 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:57.794542 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:57.795607 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:57.796728 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:57.798247 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:57.798821 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:57.798999 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:57.800489 (+ 1490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:57.800494 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:57.800498 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:15:57.800515 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:57.800546 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:15:57.800559 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:57.800654 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:57.801017 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 23:15:57.801110 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 23:15:57.802119 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.191.195:0
--local_ip_for_outbound_sockets=127.10.191.195
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:44105
--builtin_ntp_servers=127.10.191.212:44161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 23:15:57.802544 31215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:57.791523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:44105 (local address 127.10.191.194:50921)
0620 23:15:57.792389 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:57.792426 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:57.794050 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:57.794731 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:57.794739 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:57.794820 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:57.795443 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:57.795460 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:57.796883 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:57.796889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:57.798100 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:57.798112 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:57.799083 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:57.799090 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:57.799395 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:57.800158 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:15:57.800164 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:57.800218 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:15:57.800298 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:57.800655 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:57.800668 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:57.800678 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 23:15:57.800870 (+ 192us) client_negotiation.cc:240] Negotiation successful
0620 23:15:57.801243 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":261,"threads_started":1}
I20260620 23:15:57.803933 31213 heartbeater.cc:347] Connected to a master server at 127.10.191.254:44105
I20260620 23:15:57.804239 31213 heartbeater.cc:464] Registering TS with master...
I20260620 23:15:57.804962 31213 heartbeater.cc:511] Master 127.10.191.254:44105 requested a full tablet report, sending...
I20260620 23:15:57.806272 30896 ts_manager.cc:194] Registered new tserver with Master: 5b0cccf352be4a99872ad069cd705457 (127.10.191.194:42457)
I20260620 23:15:57.807176 30896 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.191.194:50921
W20260620 23:15:58.126422 31216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:15:58.126677 31216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:15:58.126809 31216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:15:58.134084 31216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:15:58.134225 31216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:15:58.134402 31216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.191.195
I20260620 23:15:58.143544 31216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:44161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.191.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.191.195
--webserver_port=0
--tserver_master_addrs=127.10.191.254:44105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.191.195
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:15:58.144845 31216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:15:58.146641 31216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:15:58.159885 31221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:58.159830 31222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:15:58.159907 31224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:15:58.160772 31216 server_base.cc:1034] running on GCE node
I20260620 23:15:58.161420 31216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:15:58.162374 31216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:15:58.163599 31216 hybrid_clock.cc:648] HybridClock initialized: now 1781997358163560 us; error 54 us; skew 500 ppm
I20260620 23:15:58.163916 31216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:15:58.166055 31216 webserver.cc:460] Webserver started at http://127.10.191.195:37491/ using document root <none> and password file <none>
I20260620 23:15:58.166729 31216 fs_manager.cc:359] Metadata directory not provided
I20260620 23:15:58.166899 31216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:15:58.167228 31216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:15:58.169963 31216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-2/data/instance:
uuid: "baefd322b82d4b02a3b2a949a3808c7a"
format_stamp: "Formatted at 2026-06-20 23:15:58 on dist-test-slave-jkbq"
server_key: "d190ee2c544b1600fe87bc131301ea24"
server_key_iv: "0b9b1c8b6a5c07581dd59a4acc9c8aa4"
server_key_version: "encryptionkey@0"
I20260620 23:15:58.170728 31216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance:
uuid: "baefd322b82d4b02a3b2a949a3808c7a"
format_stamp: "Formatted at 2026-06-20 23:15:58 on dist-test-slave-jkbq"
server_key: "d190ee2c544b1600fe87bc131301ea24"
server_key_iv: "0b9b1c8b6a5c07581dd59a4acc9c8aa4"
server_key_version: "encryptionkey@0"
I20260620 23:15:58.175781 31216 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260620 23:15:58.179047 31230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:58.180217 31216 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260620 23:15:58.180392 31216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-2/wal
uuid: "baefd322b82d4b02a3b2a949a3808c7a"
format_stamp: "Formatted at 2026-06-20 23:15:58 on dist-test-slave-jkbq"
server_key: "d190ee2c544b1600fe87bc131301ea24"
server_key_iv: "0b9b1c8b6a5c07581dd59a4acc9c8aa4"
server_key_version: "encryptionkey@0"
I20260620 23:15:58.180608 31216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:15:58.202785 31216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:15:58.203600 31216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:15:58.203907 31216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:15:58.204828 31216 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:15:58.206245 31216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:15:58.206379 31216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:58.206511 31216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:15:58.206601 31216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:15:58.239750 31216 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.195:46395
I20260620 23:15:58.239847 31342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.195:46395 every 8 connection(s)
I20260620 23:15:58.241068 31216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-2/data/info.pb
I20260620 23:15:58.248908 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 31216
I20260620 23:15:58.249094 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-2/wal/instance
I20260620 23:15:58.249529 11007 external_mini_cluster.cc:1428] Setting key fbbac4067e613c2ad4ad9639392bc00e
I20260620 23:15:58.254106 31084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:58.243655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.195:43973 (local address 127.10.191.254:44105)
0620 23:15:58.243830 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:58.243835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:58.245202 (+ 1367us) server_negotiation.cc:407] Connection header received
0620 23:15:58.247078 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:58.247083 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:58.247175 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:58.247286 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:58.248361 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:58.249155 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:58.251274 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:58.251802 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:58.251940 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:58.253325 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:58.253330 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:58.253334 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:15:58.253352 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:58.253383 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:15:58.253394 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:58.253463 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:58.253753 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 23:15:58.253846 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 23:15:58.255062 31345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:58.244186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:44105 (local address 127.10.191.195:43973)
0620 23:15:58.245039 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:58.245073 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:58.246752 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:58.247535 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:58.247543 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:58.247635 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:15:58.248167 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:58.248183 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:58.249328 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:58.249341 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:58.251103 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:58.251121 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:58.252029 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:58.252036 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:58.252360 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:58.253037 (+ 677us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:15:58.253039 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:15:58.253081 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:15:58.253161 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:15:58.253476 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:15:58.253484 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:15:58.253489 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:15:58.253627 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 23:15:58.253916 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":257,"threads_started":1}
I20260620 23:15:58.256409 31343 heartbeater.cc:347] Connected to a master server at 127.10.191.254:44105
I20260620 23:15:58.256662 31343 heartbeater.cc:464] Registering TS with master...
I20260620 23:15:58.257336 31343 heartbeater.cc:511] Master 127.10.191.254:44105 requested a full tablet report, sending...
I20260620 23:15:58.258632 30896 ts_manager.cc:194] Registered new tserver with Master: baefd322b82d4b02a3b2a949a3808c7a (127.10.191.195:46395)
I20260620 23:15:58.259485 30896 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.191.195:43973
I20260620 23:15:58.264423 11007 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:15:58.274168 31084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:58.268585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34222 (local address 127.10.191.254:44105)
0620 23:15:58.268784 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:58.268791 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:58.269237 (+ 446us) server_negotiation.cc:407] Connection header received
0620 23:15:58.269367 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:58.269371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:58.269466 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:58.269556 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:58.269994 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:58.270654 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:58.272171 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:58.272588 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:58.272710 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:58.273619 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:58.273623 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:58.273627 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:15:58.273642 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:58.273673 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:15:58.273685 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:58.273752 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:58.273865 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 23:15:58.273933 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 23:15:58.277820 30896 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34222:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:15:58.287484 30896 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:15:58.323150 31355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:58.299260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.193:42789 (local address 127.0.0.1:41382)
0620 23:15:58.313783 (+ 14523us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:58.313836 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:58.314068 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:58.315990 (+ 1922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:58.316001 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:58.316064 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:58.316638 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:58.316649 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:58.318684 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:58.318692 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:58.320531 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:58.320543 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:58.322053 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:58.322059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:58.322341 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:58.322400 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 23:15:58.322565 (+ 165us) client_negotiation.cc:240] Negotiation successful
0620 23:15:58.322756 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12331,"mutex_wait_us":3,"thread_start_us":13504,"threads_started":1}
I20260620 23:15:58.323208 31356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:58.299493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41382 (local address 127.10.191.193:42789)
0620 23:15:58.314841 (+ 15348us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:58.314855 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:58.314914 (+ 59us) server_negotiation.cc:407] Connection header received
0620 23:15:58.315044 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:58.315056 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:58.315406 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:58.316270 (+ 864us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:58.316877 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:58.318553 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:58.320779 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:58.321844 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:58.322058 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:58.322297 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:58.322622 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 23:15:58.322802 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15112,"thread_start_us":16153,"threads_started":1}
I20260620 23:15:58.326692 31017 tablet_service.cc:1467] Processing CreateTablet for tablet 395f6900c77f4ca5aaa38fed9e88a0d8 (DEFAULT_TABLE table=test-table [id=12f62eca45d64152a0b00296ffa50826]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:15:58.327795 31359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:58.300824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41216 (local address 127.10.191.195:46395)
0620 23:15:58.315942 (+ 15118us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:58.315953 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:58.317111 (+ 1158us) server_negotiation.cc:407] Connection header received
0620 23:15:58.317271 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:58.317282 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:58.317535 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:58.317714 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:58.318421 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:58.319960 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:58.322598 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:58.326729 (+ 4131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:58.326972 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:58.327174 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:58.327291 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 23:15:58.327447 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14909,"thread_start_us":14743,"threads_started":1}
I20260620 23:15:58.328105 31358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:58.299574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.195:46395 (local address 127.0.0.1:41216)
0620 23:15:58.316985 (+ 17411us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:58.317002 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:58.317133 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:58.317978 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:58.317983 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:58.318005 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:58.318256 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:58.318263 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:58.320127 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:58.320132 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:58.322308 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:58.322322 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:58.326938 (+ 4616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:58.326946 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:58.327116 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:58.327148 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 23:15:58.327255 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 23:15:58.327393 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17316,"mutex_wait_us":23,"thread_start_us":17754,"threads_started":1}
I20260620 23:15:58.328145 31017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 395f6900c77f4ca5aaa38fed9e88a0d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:15:58.332145 31278 tablet_service.cc:1467] Processing CreateTablet for tablet 395f6900c77f4ca5aaa38fed9e88a0d8 (DEFAULT_TABLE table=test-table [id=12f62eca45d64152a0b00296ffa50826]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:15:58.332978 31360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:58.301416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33058 (local address 127.10.191.194:42457)
0620 23:15:58.315405 (+ 13989us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:58.315426 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:58.317670 (+ 2244us) server_negotiation.cc:407] Connection header received
0620 23:15:58.318010 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:58.318021 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:58.318219 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:58.318389 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:58.326770 (+ 8381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:58.328268 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:58.330337 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:58.331778 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:58.331943 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:58.332142 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:58.332369 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 23:15:58.332537 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13839,"thread_start_us":13722,"threads_started":1}
I20260620 23:15:58.333086 31357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:58.300061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.194:42457 (local address 127.0.0.1:33058)
0620 23:15:58.317557 (+ 17496us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:58.317580 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:58.317765 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:58.318441 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:58.318445 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:58.318470 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:58.326569 (+ 8099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:58.326578 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:58.328455 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:58.328459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:58.330127 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:58.330142 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:58.331983 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:58.331990 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:58.332167 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:58.332203 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 23:15:58.332306 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 23:15:58.332449 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17387,"thread_start_us":16079,"threads_started":1}
I20260620 23:15:58.334012 31278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 395f6900c77f4ca5aaa38fed9e88a0d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:15:58.336895 31148 tablet_service.cc:1467] Processing CreateTablet for tablet 395f6900c77f4ca5aaa38fed9e88a0d8 (DEFAULT_TABLE table=test-table [id=12f62eca45d64152a0b00296ffa50826]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:15:58.338244 31148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 395f6900c77f4ca5aaa38fed9e88a0d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:15:58.341334 31361 tablet_bootstrap.cc:492] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 0a407a37f91344899ee365cdaae9e562: Bootstrap starting.
I20260620 23:15:58.344195 31082 heartbeater.cc:503] Master 127.10.191.254:44105 was elected leader, sending a full tablet report...
I20260620 23:15:58.344791 31361 tablet_bootstrap.cc:654] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 0a407a37f91344899ee365cdaae9e562: Neither blocks nor log segments found. Creating new log.
I20260620 23:15:58.346467 31361 log.cc:824] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 0a407a37f91344899ee365cdaae9e562: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:58.351030 31363 tablet_bootstrap.cc:492] T 395f6900c77f4ca5aaa38fed9e88a0d8 P baefd322b82d4b02a3b2a949a3808c7a: Bootstrap starting.
I20260620 23:15:58.353049 31364 tablet_bootstrap.cc:492] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 5b0cccf352be4a99872ad069cd705457: Bootstrap starting.
I20260620 23:15:58.354398 31361 tablet_bootstrap.cc:492] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 0a407a37f91344899ee365cdaae9e562: No bootstrap required, opened a new log
I20260620 23:15:58.354687 31361 ts_tablet_manager.cc:1397] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 0a407a37f91344899ee365cdaae9e562: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260620 23:15:58.355458 31363 tablet_bootstrap.cc:654] T 395f6900c77f4ca5aaa38fed9e88a0d8 P baefd322b82d4b02a3b2a949a3808c7a: Neither blocks nor log segments found. Creating new log.
I20260620 23:15:58.355985 31364 tablet_bootstrap.cc:654] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 5b0cccf352be4a99872ad069cd705457: Neither blocks nor log segments found. Creating new log.
I20260620 23:15:58.356681 31363 log.cc:824] T 395f6900c77f4ca5aaa38fed9e88a0d8 P baefd322b82d4b02a3b2a949a3808c7a: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:58.357187 31364 log.cc:824] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 5b0cccf352be4a99872ad069cd705457: Log is configured to *not* fsync() on all Append() calls
I20260620 23:15:58.359997 31363 tablet_bootstrap.cc:492] T 395f6900c77f4ca5aaa38fed9e88a0d8 P baefd322b82d4b02a3b2a949a3808c7a: No bootstrap required, opened a new log
I20260620 23:15:58.360229 31364 tablet_bootstrap.cc:492] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 5b0cccf352be4a99872ad069cd705457: No bootstrap required, opened a new log
I20260620 23:15:58.360261 31363 ts_tablet_manager.cc:1397] T 395f6900c77f4ca5aaa38fed9e88a0d8 P baefd322b82d4b02a3b2a949a3808c7a: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 23:15:58.360482 31364 ts_tablet_manager.cc:1397] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 5b0cccf352be4a99872ad069cd705457: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260620 23:15:58.365475 31361 raft_consensus.cc:348] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 0a407a37f91344899ee365cdaae9e562 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a407a37f91344899ee365cdaae9e562" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 42789 } } peers { permanent_uuid: "baefd322b82d4b02a3b2a949a3808c7a" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46395 } } peers { permanent_uuid: "5b0cccf352be4a99872ad069cd705457" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 42457 } }
I20260620 23:15:58.365936 31361 raft_consensus.cc:374] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 0a407a37f91344899ee365cdaae9e562 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:15:58.366035 31361 raft_consensus.cc:729] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 0a407a37f91344899ee365cdaae9e562 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a407a37f91344899ee365cdaae9e562, State: Initialized, Role: FOLLOWER
I20260620 23:15:58.366679 31361 consensus_queue.cc:260] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 0a407a37f91344899ee365cdaae9e562 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a407a37f91344899ee365cdaae9e562" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 42789 } } peers { permanent_uuid: "baefd322b82d4b02a3b2a949a3808c7a" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46395 } } peers { permanent_uuid: "5b0cccf352be4a99872ad069cd705457" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 42457 } }
I20260620 23:15:58.367712 31363 raft_consensus.cc:348] T 395f6900c77f4ca5aaa38fed9e88a0d8 P baefd322b82d4b02a3b2a949a3808c7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a407a37f91344899ee365cdaae9e562" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 42789 } } peers { permanent_uuid: "baefd322b82d4b02a3b2a949a3808c7a" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46395 } } peers { permanent_uuid: "5b0cccf352be4a99872ad069cd705457" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 42457 } }
I20260620 23:15:58.368125 31363 raft_consensus.cc:374] T 395f6900c77f4ca5aaa38fed9e88a0d8 P baefd322b82d4b02a3b2a949a3808c7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:15:58.368225 31363 raft_consensus.cc:729] T 395f6900c77f4ca5aaa38fed9e88a0d8 P baefd322b82d4b02a3b2a949a3808c7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baefd322b82d4b02a3b2a949a3808c7a, State: Initialized, Role: FOLLOWER
I20260620 23:15:58.368784 31363 consensus_queue.cc:260] T 395f6900c77f4ca5aaa38fed9e88a0d8 P baefd322b82d4b02a3b2a949a3808c7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a407a37f91344899ee365cdaae9e562" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 42789 } } peers { permanent_uuid: "baefd322b82d4b02a3b2a949a3808c7a" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46395 } } peers { permanent_uuid: "5b0cccf352be4a99872ad069cd705457" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 42457 } }
I20260620 23:15:58.369829 31343 heartbeater.cc:503] Master 127.10.191.254:44105 was elected leader, sending a full tablet report...
I20260620 23:15:58.370543 31363 ts_tablet_manager.cc:1428] T 395f6900c77f4ca5aaa38fed9e88a0d8 P baefd322b82d4b02a3b2a949a3808c7a: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20260620 23:15:58.372227 31364 raft_consensus.cc:348] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 5b0cccf352be4a99872ad069cd705457 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a407a37f91344899ee365cdaae9e562" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 42789 } } peers { permanent_uuid: "baefd322b82d4b02a3b2a949a3808c7a" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46395 } } peers { permanent_uuid: "5b0cccf352be4a99872ad069cd705457" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 42457 } }
I20260620 23:15:58.372751 31364 raft_consensus.cc:374] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 5b0cccf352be4a99872ad069cd705457 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:15:58.372884 31364 raft_consensus.cc:729] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 5b0cccf352be4a99872ad069cd705457 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b0cccf352be4a99872ad069cd705457, State: Initialized, Role: FOLLOWER
I20260620 23:15:58.373587 31364 consensus_queue.cc:260] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 5b0cccf352be4a99872ad069cd705457 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a407a37f91344899ee365cdaae9e562" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 42789 } } peers { permanent_uuid: "baefd322b82d4b02a3b2a949a3808c7a" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46395 } } peers { permanent_uuid: "5b0cccf352be4a99872ad069cd705457" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 42457 } }
I20260620 23:15:58.374778 31213 heartbeater.cc:503] Master 127.10.191.254:44105 was elected leader, sending a full tablet report...
I20260620 23:15:58.375556 31361 ts_tablet_manager.cc:1428] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 0a407a37f91344899ee365cdaae9e562: Time spent starting tablet: real 0.021s user 0.017s sys 0.004s
I20260620 23:15:58.376200 31364 ts_tablet_manager.cc:1428] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 5b0cccf352be4a99872ad069cd705457: Time spent starting tablet: real 0.016s user 0.012s sys 0.003s
W20260620 23:15:58.492745 31344 tablet.cc:2369] T 395f6900c77f4ca5aaa38fed9e88a0d8 P baefd322b82d4b02a3b2a949a3808c7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:15:58.498358 31367 raft_consensus.cc:482] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 0a407a37f91344899ee365cdaae9e562 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:15:58.498703 31367 raft_consensus.cc:504] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 0a407a37f91344899ee365cdaae9e562 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a407a37f91344899ee365cdaae9e562" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 42789 } } peers { permanent_uuid: "baefd322b82d4b02a3b2a949a3808c7a" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46395 } } peers { permanent_uuid: "5b0cccf352be4a99872ad069cd705457" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 42457 } }
I20260620 23:15:58.500408 31367 leader_election.cc:288] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 0a407a37f91344899ee365cdaae9e562 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers baefd322b82d4b02a3b2a949a3808c7a (127.10.191.195:46395), 5b0cccf352be4a99872ad069cd705457 (127.10.191.194:42457)
I20260620 23:15:58.507066 31359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:58.500790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:42821 (local address 127.10.191.195:46395)
0620 23:15:58.501007 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:58.501015 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:58.501817 (+ 802us) server_negotiation.cc:407] Connection header received
0620 23:15:58.501973 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:58.501977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:58.502031 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:58.502145 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:58.502793 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:58.503562 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:58.505436 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:58.506167 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:58.506331 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:58.506410 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:58.506690 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 23:15:58.506807 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 23:15:58.507071 31360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:58.500861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.191.193:51465 (local address 127.10.191.194:42457)
0620 23:15:58.501101 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:58.501109 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:58.501643 (+ 534us) server_negotiation.cc:407] Connection header received
0620 23:15:58.501830 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:58.501837 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:58.501935 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:58.502049 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:15:58.502672 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:58.503456 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:58.505413 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:58.506109 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:58.506289 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:58.506357 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:58.506669 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 23:15:58.506786 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 23:15:58.507182 31370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:58.500799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.195:46395 (local address 127.10.191.193:42821)
0620 23:15:58.501688 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:58.501708 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:58.501852 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:58.502158 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:58.502163 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:58.502189 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:58.502622 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:58.502631 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:58.503700 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:58.503707 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:58.505219 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:58.505231 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:58.506359 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:58.506365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:58.506539 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:58.506561 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 23:15:58.506663 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 23:15:58.506787 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"mutex_wait_us":7,"thread_start_us":378,"threads_started":1}
I20260620 23:15:58.507604 31371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:58.500690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.194:42457 (local address 127.10.191.193:51465)
0620 23:15:58.501526 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0620 23:15:58.501542 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:15:58.501662 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:15:58.502074 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:15:58.502078 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:15:58.502101 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:15:58.502518 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:58.502528 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:58.503579 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:58.503587 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:58.505214 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:15:58.505226 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:15:58.506299 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:15:58.506305 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:15:58.506484 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:58.506532 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 23:15:58.507098 (+ 566us) client_negotiation.cc:240] Negotiation successful
0620 23:15:58.507235 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"mutex_wait_us":107,"spinlock_wait_cycles":20352,"thread_start_us":660,"threads_started":1}
I20260620 23:15:58.508456 31298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "395f6900c77f4ca5aaa38fed9e88a0d8" candidate_uuid: "0a407a37f91344899ee365cdaae9e562" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baefd322b82d4b02a3b2a949a3808c7a" is_pre_election: true
I20260620 23:15:58.508740 31168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "395f6900c77f4ca5aaa38fed9e88a0d8" candidate_uuid: "0a407a37f91344899ee365cdaae9e562" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b0cccf352be4a99872ad069cd705457" is_pre_election: true
I20260620 23:15:58.509048 31298 raft_consensus.cc:2454] T 395f6900c77f4ca5aaa38fed9e88a0d8 P baefd322b82d4b02a3b2a949a3808c7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a407a37f91344899ee365cdaae9e562 in term 0.
I20260620 23:15:58.509241 31168 raft_consensus.cc:2454] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 5b0cccf352be4a99872ad069cd705457 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a407a37f91344899ee365cdaae9e562 in term 0.
I20260620 23:15:58.509712 30973 leader_election.cc:302] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 0a407a37f91344899ee365cdaae9e562 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a407a37f91344899ee365cdaae9e562, baefd322b82d4b02a3b2a949a3808c7a; no voters:
I20260620 23:15:58.510095 31367 raft_consensus.cc:2781] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 0a407a37f91344899ee365cdaae9e562 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:15:58.510263 31367 raft_consensus.cc:482] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 0a407a37f91344899ee365cdaae9e562 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:15:58.510362 31367 raft_consensus.cc:3037] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 0a407a37f91344899ee365cdaae9e562 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:15:58.512539 31367 raft_consensus.cc:504] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 0a407a37f91344899ee365cdaae9e562 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a407a37f91344899ee365cdaae9e562" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 42789 } } peers { permanent_uuid: "baefd322b82d4b02a3b2a949a3808c7a" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46395 } } peers { permanent_uuid: "5b0cccf352be4a99872ad069cd705457" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 42457 } }
I20260620 23:15:58.513123 31367 leader_election.cc:288] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 0a407a37f91344899ee365cdaae9e562 [CANDIDATE]: Term 1 election: Requested vote from peers baefd322b82d4b02a3b2a949a3808c7a (127.10.191.195:46395), 5b0cccf352be4a99872ad069cd705457 (127.10.191.194:42457)
I20260620 23:15:58.513643 31298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "395f6900c77f4ca5aaa38fed9e88a0d8" candidate_uuid: "0a407a37f91344899ee365cdaae9e562" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baefd322b82d4b02a3b2a949a3808c7a"
I20260620 23:15:58.513643 31168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "395f6900c77f4ca5aaa38fed9e88a0d8" candidate_uuid: "0a407a37f91344899ee365cdaae9e562" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b0cccf352be4a99872ad069cd705457"
I20260620 23:15:58.513845 31298 raft_consensus.cc:3037] T 395f6900c77f4ca5aaa38fed9e88a0d8 P baefd322b82d4b02a3b2a949a3808c7a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:15:58.513845 31168 raft_consensus.cc:3037] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 5b0cccf352be4a99872ad069cd705457 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:15:58.517086 31298 raft_consensus.cc:2454] T 395f6900c77f4ca5aaa38fed9e88a0d8 P baefd322b82d4b02a3b2a949a3808c7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a407a37f91344899ee365cdaae9e562 in term 1.
I20260620 23:15:58.517102 31168 raft_consensus.cc:2454] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 5b0cccf352be4a99872ad069cd705457 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a407a37f91344899ee365cdaae9e562 in term 1.
I20260620 23:15:58.517532 30972 leader_election.cc:302] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 0a407a37f91344899ee365cdaae9e562 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0a407a37f91344899ee365cdaae9e562, 5b0cccf352be4a99872ad069cd705457, baefd322b82d4b02a3b2a949a3808c7a; no voters:
I20260620 23:15:58.517905 31367 raft_consensus.cc:2781] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 0a407a37f91344899ee365cdaae9e562 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:15:58.518365 31367 raft_consensus.cc:686] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 0a407a37f91344899ee365cdaae9e562 [term 1 LEADER]: Becoming Leader. State: Replica: 0a407a37f91344899ee365cdaae9e562, State: Running, Role: LEADER
I20260620 23:15:58.518950 31367 consensus_queue.cc:237] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 0a407a37f91344899ee365cdaae9e562 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a407a37f91344899ee365cdaae9e562" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 42789 } } peers { permanent_uuid: "baefd322b82d4b02a3b2a949a3808c7a" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46395 } } peers { permanent_uuid: "5b0cccf352be4a99872ad069cd705457" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 42457 } }
I20260620 23:15:58.523312 30896 catalog_manager.cc:5495] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 0a407a37f91344899ee365cdaae9e562 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a407a37f91344899ee365cdaae9e562 (127.10.191.193). New cstate: current_term: 1 leader_uuid: "0a407a37f91344899ee365cdaae9e562" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a407a37f91344899ee365cdaae9e562" member_type: VOTER last_known_addr { host: "127.10.191.193" port: 42789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "baefd322b82d4b02a3b2a949a3808c7a" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b0cccf352be4a99872ad069cd705457" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 42457 } health_report { overall_health: UNKNOWN } } }
W20260620 23:15:58.540410 31214 tablet.cc:2369] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 5b0cccf352be4a99872ad069cd705457: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:15:58.565915 31356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:58.560472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41388 (local address 127.10.191.193:42789)
0620 23:15:58.560691 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:58.560697 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:58.560799 (+ 102us) server_negotiation.cc:407] Connection header received
0620 23:15:58.561048 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:58.561053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:58.561143 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:58.561298 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:15:58.561665 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:58.562386 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:58.563808 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:58.564214 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:58.564345 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:58.564847 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:15:58.565352 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:15:58.565521 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:58.565571 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 23:15:58.565651 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260620 23:15:58.572849 31083 tablet.cc:2369] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 0a407a37f91344899ee365cdaae9e562: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:15:58.574229 31298 raft_consensus.cc:1261] T 395f6900c77f4ca5aaa38fed9e88a0d8 P baefd322b82d4b02a3b2a949a3808c7a [term 1 FOLLOWER]: Refusing update from remote peer 0a407a37f91344899ee365cdaae9e562: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:15:58.574234 31168 raft_consensus.cc:1261] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 5b0cccf352be4a99872ad069cd705457 [term 1 FOLLOWER]: Refusing update from remote peer 0a407a37f91344899ee365cdaae9e562: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:15:58.575227 31372 consensus_queue.cc:1035] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 0a407a37f91344899ee365cdaae9e562 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baefd322b82d4b02a3b2a949a3808c7a" member_type: VOTER last_known_addr { host: "127.10.191.195" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:15:58.575549 31367 consensus_queue.cc:1035] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 0a407a37f91344899ee365cdaae9e562 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b0cccf352be4a99872ad069cd705457" member_type: VOTER last_known_addr { host: "127.10.191.194" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:15:58.587373 31376 mvcc.cc:205] Tried to move back new op lower bound from 7299061180709867520 to 7299061180497166336. Current Snapshot: MvccSnapshot[applied={T|T < 7299061180709867520}]
I20260620 23:15:58.590101 31377 mvcc.cc:205] Tried to move back new op lower bound from 7299061180709867520 to 7299061180497166336. Current Snapshot: MvccSnapshot[applied={T|T < 7299061180709867520}]
I20260620 23:15:58.599327 30895 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:15:58.599632 30895 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:15:58.603411 30895 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e537d5211dc469c9ffce23243b6c1ee: Sending DeleteTablet for 3 replicas of tablet 395f6900c77f4ca5aaa38fed9e88a0d8
I20260620 23:15:58.604470 31017 tablet_service.cc:1514] Processing DeleteTablet for tablet 395f6900c77f4ca5aaa38fed9e88a0d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:15:58 UTC) from {username='slave'} at 127.0.0.1:41382
I20260620 23:15:58.604573 31278 tablet_service.cc:1514] Processing DeleteTablet for tablet 395f6900c77f4ca5aaa38fed9e88a0d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:15:58 UTC) from {username='slave'} at 127.0.0.1:41216
I20260620 23:15:58.604643 31148 tablet_service.cc:1514] Processing DeleteTablet for tablet 395f6900c77f4ca5aaa38fed9e88a0d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:15:58 UTC) from {username='slave'} at 127.0.0.1:33058
I20260620 23:15:58.605372 31385 tablet_replica.cc:330] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 0a407a37f91344899ee365cdaae9e562: stopping tablet replica
I20260620 23:15:58.605722 31385 raft_consensus.cc:2229] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 0a407a37f91344899ee365cdaae9e562 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:15:58.606225 31385 raft_consensus.cc:2258] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 0a407a37f91344899ee365cdaae9e562 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:15:58.608093 31385 ts_tablet_manager.cc:1905] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 0a407a37f91344899ee365cdaae9e562: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:15:58.611742 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 30955
I20260620 23:15:58.616144 31385 ts_tablet_manager.cc:1918] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 0a407a37f91344899ee365cdaae9e562: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:15:58.616328 31385 log.cc:1196] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 0a407a37f91344899ee365cdaae9e562: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-0/wal/wals/395f6900c77f4ca5aaa38fed9e88a0d8
I20260620 23:15:58.617017 31385 ts_tablet_manager.cc:1939] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 0a407a37f91344899ee365cdaae9e562: Deleting consensus metadata
I20260620 23:15:58.617957 31386 tablet_replica.cc:330] T 395f6900c77f4ca5aaa38fed9e88a0d8 P baefd322b82d4b02a3b2a949a3808c7a: stopping tablet replica
I20260620 23:15:58.618403 31386 raft_consensus.cc:2229] T 395f6900c77f4ca5aaa38fed9e88a0d8 P baefd322b82d4b02a3b2a949a3808c7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:15:58.618943 31386 raft_consensus.cc:2258] T 395f6900c77f4ca5aaa38fed9e88a0d8 P baefd322b82d4b02a3b2a949a3808c7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:15:58.618968 30880 catalog_manager.cc:4841] TS 0a407a37f91344899ee365cdaae9e562 (127.10.191.193:42789): tablet 395f6900c77f4ca5aaa38fed9e88a0d8 (table test-table [id=12f62eca45d64152a0b00296ffa50826]) successfully deleted
I20260620 23:15:58.619735 31387 tablet_replica.cc:330] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 5b0cccf352be4a99872ad069cd705457: stopping tablet replica
I20260620 23:15:58.620133 31387 raft_consensus.cc:2229] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 5b0cccf352be4a99872ad069cd705457 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:15:58.620601 31387 raft_consensus.cc:2258] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 5b0cccf352be4a99872ad069cd705457 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:15:58.622949 31387 ts_tablet_manager.cc:1905] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 5b0cccf352be4a99872ad069cd705457: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:15:58.623812 31386 ts_tablet_manager.cc:1905] T 395f6900c77f4ca5aaa38fed9e88a0d8 P baefd322b82d4b02a3b2a949a3808c7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:15:58.628245 31356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:58.612854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41400 (local address 127.10.191.193:42789)
0620 23:15:58.613022 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:58.613027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:58.613743 (+ 716us) server_negotiation.cc:407] Connection header received
0620 23:15:58.613945 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:58.613953 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:58.614074 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:58.614200 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:58.618354 (+ 4154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:58.619565 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:58.621080 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:58.621525 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:58.621601 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:58.626517 (+ 4916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:58.626528 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:58.626544 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:15:58.627349 (+ 805us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:58.627404 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:15:58.627431 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:58.627539 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:58.627768 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 23:15:58.627880 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 23:15:58.629673 31077 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:15:58.629762 31386 ts_tablet_manager.cc:1918] T 395f6900c77f4ca5aaa38fed9e88a0d8 P baefd322b82d4b02a3b2a949a3808c7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:15:58.629823 31387 ts_tablet_manager.cc:1918] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 5b0cccf352be4a99872ad069cd705457: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:15:58.629904 31386 log.cc:1196] T 395f6900c77f4ca5aaa38fed9e88a0d8 P baefd322b82d4b02a3b2a949a3808c7a: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-2/wal/wals/395f6900c77f4ca5aaa38fed9e88a0d8
I20260620 23:15:58.629983 31387 log.cc:1196] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 5b0cccf352be4a99872ad069cd705457: Deleting WAL directory at /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781997042893763-11007-0/minicluster-data/ts-1/wal/wals/395f6900c77f4ca5aaa38fed9e88a0d8
I20260620 23:15:58.630538 31387 ts_tablet_manager.cc:1939] T 395f6900c77f4ca5aaa38fed9e88a0d8 P 5b0cccf352be4a99872ad069cd705457: Deleting consensus metadata
I20260620 23:15:58.630568 31386 ts_tablet_manager.cc:1939] T 395f6900c77f4ca5aaa38fed9e88a0d8 P baefd322b82d4b02a3b2a949a3808c7a: Deleting consensus metadata
I20260620 23:15:58.632606 30883 catalog_manager.cc:4841] TS baefd322b82d4b02a3b2a949a3808c7a (127.10.191.195:46395): tablet 395f6900c77f4ca5aaa38fed9e88a0d8 (table test-table [id=12f62eca45d64152a0b00296ffa50826]) successfully deleted
I20260620 23:15:58.632746 30882 catalog_manager.cc:4841] TS 5b0cccf352be4a99872ad069cd705457 (127.10.191.194:42457): tablet 395f6900c77f4ca5aaa38fed9e88a0d8 (table test-table [id=12f62eca45d64152a0b00296ffa50826]) successfully deleted
I20260620 23:15:58.725376 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 30955
I20260620 23:15:58.743429 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 31086
I20260620 23:15:58.750222 31360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:58.744280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33068 (local address 127.10.191.194:42457)
0620 23:15:58.744462 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:58.744466 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:58.744484 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:15:58.744683 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:58.744688 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:58.744790 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:58.744893 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:58.745252 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:58.745988 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:58.747278 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:58.747679 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:58.747826 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:58.748865 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:58.748873 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:58.748887 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:15:58.749473 (+ 586us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:58.749517 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:15:58.749536 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:58.749670 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:58.749807 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 23:15:58.749909 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 23:15:58.751039 31208 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:15:58.838779 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 31086
I20260620 23:15:58.856041 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 31216
I20260620 23:15:58.862478 31359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:15:58.856853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41218 (local address 127.10.191.195:46395)
0620 23:15:58.857043 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 23:15:58.857048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:15:58.857066 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:15:58.857241 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:15:58.857245 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:15:58.857329 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:15:58.857423 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:15:58.857780 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:58.858489 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:58.859754 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:15:58.860131 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:15:58.860223 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:15:58.861170 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:15:58.861178 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:15:58.861194 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:15:58.861736 (+ 542us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:15:58.861780 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:15:58.861798 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:15:58.861904 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 23:15:58.862047 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 23:15:58.862142 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 23:15:58.863176 31338 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:15:58.951000 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 31216
I20260620 23:15:58.969272 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 30865
I20260620 23:15:58.970353 30926 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:15:59.077595 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 30865
2026-06-20T23:15:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2787 ms)
[----------] 6 tests from EncryptionPolicyTest (21525 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997042893763-11007-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:15:59 dist-test-slave-jkbq krb5kdc[31395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:15:59 dist-test-slave-jkbq krb5kdc[31395](info): set up 2 sockets
Jun 20 23:15:59 dist-test-slave-jkbq krb5kdc[31395](info): commencing operation
krb5kdc: starting...
W20260620 23:16:00.183284 11007 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Jun 20 23:16:00 dist-test-slave-jkbq krb5kdc[31395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:16:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:16:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:16:00.430414 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34687
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:16:00.752279 31408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:00.752579 31408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:00.752779 31408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:00.759775 31408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:16:00.759876 31408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:00.759985 31408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:00.760093 31408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:16:00.760186 31408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:16:00.770292 31408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34687
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:16:00.771575 31408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:00.772883 31408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:00.785128 31416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:00.785183 31414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:00.785614 31413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:00.787230 31408 server_base.cc:1034] running on GCE node
I20260620 23:16:00.788543 31408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:00.790695 31408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:00.791965 31408 hybrid_clock.cc:648] HybridClock initialized: now 1781997360791842 us; error 94 us; skew 500 ppm
I20260620 23:16:00.792474 31408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:16:00 dist-test-slave-jkbq krb5kdc[31395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997360, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:00.796350 31408 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:16:00.798166 31408 webserver.cc:460] Webserver started at http://127.0.0.1:41539/ using document root <none> and password file <none>
I20260620 23:16:00.799068 31408 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:00.799224 31408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:00.799582 31408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:00.803303 31408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "6cb4853066084fda83eb095a72aa0878"
format_stamp: "Formatted at 2026-06-20 23:16:00 on dist-test-slave-jkbq"
server_key: "96864fb8fe3cd82dbf65a114e5c6b018"
server_key_iv: "53e03ec49206e1eab98e833ce951b1a6"
server_key_version: "encryptionkey@0"
I20260620 23:16:00.804356 31408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "6cb4853066084fda83eb095a72aa0878"
format_stamp: "Formatted at 2026-06-20 23:16:00 on dist-test-slave-jkbq"
server_key: "96864fb8fe3cd82dbf65a114e5c6b018"
server_key_iv: "53e03ec49206e1eab98e833ce951b1a6"
server_key_version: "encryptionkey@0"
I20260620 23:16:00.811453 31408 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20260620 23:16:00.815774 31423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:00.817370 31408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:16:00.817512 31408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "6cb4853066084fda83eb095a72aa0878"
format_stamp: "Formatted at 2026-06-20 23:16:00 on dist-test-slave-jkbq"
server_key: "96864fb8fe3cd82dbf65a114e5c6b018"
server_key_iv: "53e03ec49206e1eab98e833ce951b1a6"
server_key_version: "encryptionkey@0"
I20260620 23:16:00.817710 31408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:00.837646 31408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:00.841545 31408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:00.841769 31408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:00.860190 31408 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34687
I20260620 23:16:00.860214 31474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34687 every 8 connection(s)
I20260620 23:16:00.861618 31408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:16:00.863595 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 31408
I20260620 23:16:00.863816 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:16:00.864470 11007 external_mini_cluster.cc:1428] Setting key bcac6592d416f207954f8b3ecfec9a32
I20260620 23:16:00.865864 31475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 23:16:00 dist-test-slave-jkbq krb5kdc[31395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 23:16:00.879469 31475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cb4853066084fda83eb095a72aa0878: Bootstrap starting.
I20260620 23:16:00.883594 31475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cb4853066084fda83eb095a72aa0878: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:00.884867 31475 log.cc:824] T 00000000000000000000000000000000 P 6cb4853066084fda83eb095a72aa0878: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:00.887811 31475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cb4853066084fda83eb095a72aa0878: No bootstrap required, opened a new log
I20260620 23:16:00.888588 31478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:00.866482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54957 (local address 127.0.0.1:34687)
0620 23:16:00.867302 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:00.867317 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:00.867370 (+ 53us) server_negotiation.cc:407] Connection header received
0620 23:16:00.869227 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:00.869253 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:00.869775 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:00.870223 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:00.871651 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:00.873690 (+ 2039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:00.875003 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:00.876023 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:00.876208 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:00.880290 (+ 4082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:00.880300 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:00.880318 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:00.880361 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:00.884331 (+ 3970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:00.885207 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:00.885214 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:00.885220 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:00.885403 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:00.885906 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:00.885913 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:00.885916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:00.886426 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:00.886748 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:00.887300 (+ 552us) server_negotiation.cc:299] Negotiation successful
0620 23:16:00.887593 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":363,"threads_started":1}
I20260620 23:16:00.892393 11007 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:16:00.895768 31475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cb4853066084fda83eb095a72aa0878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb4853066084fda83eb095a72aa0878" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34687 } }
I20260620 23:16:00.896134 31475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cb4853066084fda83eb095a72aa0878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:00.896245 31475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cb4853066084fda83eb095a72aa0878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cb4853066084fda83eb095a72aa0878, State: Initialized, Role: FOLLOWER
I20260620 23:16:00.896787 31475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cb4853066084fda83eb095a72aa0878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb4853066084fda83eb095a72aa0878" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34687 } }
I20260620 23:16:00.897028 31475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6cb4853066084fda83eb095a72aa0878 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:16:00.897181 31475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cb4853066084fda83eb095a72aa0878 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:16:00.897389 31475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cb4853066084fda83eb095a72aa0878 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:00.899710 31475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cb4853066084fda83eb095a72aa0878 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb4853066084fda83eb095a72aa0878" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34687 } }
I20260620 23:16:00.900236 31475 leader_election.cc:302] T 00000000000000000000000000000000 P 6cb4853066084fda83eb095a72aa0878 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cb4853066084fda83eb095a72aa0878; no voters:
I20260620 23:16:00.900913 31475 leader_election.cc:288] T 00000000000000000000000000000000 P 6cb4853066084fda83eb095a72aa0878 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:16:00.901015 31485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cb4853066084fda83eb095a72aa0878 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:16:00.902660 31485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cb4853066084fda83eb095a72aa0878 [term 1 LEADER]: Becoming Leader. State: Replica: 6cb4853066084fda83eb095a72aa0878, State: Running, Role: LEADER
I20260620 23:16:00.903334 31485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cb4853066084fda83eb095a72aa0878 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb4853066084fda83eb095a72aa0878" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34687 } }
I20260620 23:16:00.903491 31475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cb4853066084fda83eb095a72aa0878 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:16:00.907001 31486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cb4853066084fda83eb095a72aa0878 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6cb4853066084fda83eb095a72aa0878" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb4853066084fda83eb095a72aa0878" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34687 } } }
I20260620 23:16:00.907385 31486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cb4853066084fda83eb095a72aa0878 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:00.912215 31487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cb4853066084fda83eb095a72aa0878 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cb4853066084fda83eb095a72aa0878. Latest consensus state: current_term: 1 leader_uuid: "6cb4853066084fda83eb095a72aa0878" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb4853066084fda83eb095a72aa0878" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34687 } } }
I20260620 23:16:00.912688 31487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cb4853066084fda83eb095a72aa0878 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:00.914741 31478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:00.897107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37229 (local address 127.0.0.1:34687)
0620 23:16:00.897283 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:00.897291 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:00.897835 (+ 544us) server_negotiation.cc:407] Connection header received
0620 23:16:00.898029 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:00.898035 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:00.898112 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:00.898251 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:00.899719 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:00.900700 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:00.902447 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:00.903122 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:00.903288 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:00.905975 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:00.905984 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:00.905988 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:00.906024 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:00.910670 (+ 4646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:00.911929 (+ 1259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:00.911938 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:00.911944 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:00.912154 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:00.912916 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:00.912920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:00.912923 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:00.913401 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:00.913721 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:00.914105 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 23:16:00.914286 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 23:16:00.919414 31491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:16:00.925055 31491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:16:00.936780 31491 catalog_manager.cc:1347] Generated new cluster ID: 4df35e60009b474b82c55d475c08ed65
I20260620 23:16:00.936895 31491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:16:00.959028 31491 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:16:00.960335 31491 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:16:00.966766 31491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6cb4853066084fda83eb095a72aa0878: Generated new TSK 0
I20260620 23:16:00.967561 31491 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:16:00.972918 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 31408
I20260620 23:16:00.974254 31470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:16:01.063226 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 31408
W20260620 23:16:01.080309 31393 connection.cc:535] client connection to 127.0.0.1:34687 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34687): Connection reset by peer (error 104)
2026-06-20T23:16:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1976 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997042893763-11007-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:16:01 dist-test-slave-jkbq krb5kdc[31507](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:16:01 dist-test-slave-jkbq krb5kdc[31507](info): set up 2 sockets
Jun 20 23:16:01 dist-test-slave-jkbq krb5kdc[31507](info): commencing operation
krb5kdc: starting...
W20260620 23:16:02.163136 11007 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Jun 20 23:16:02 dist-test-slave-jkbq krb5kdc[31507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:16:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:16:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:16:02.401325 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42059
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:16:02.724638 31520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:02.724898 31520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:02.724988 31520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:02.732185 31520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:16:02.732322 31520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:02.732424 31520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:02.732537 31520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:16:02.732614 31520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:16:02.742161 31520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42059
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:16:02.743449 31520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:02.744652 31520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:02.755967 31525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:02.756166 31526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:02.756348 31528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:02.757350 31520 server_base.cc:1034] running on GCE node
I20260620 23:16:02.758148 31520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:02.759586 31520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:02.760825 31520 hybrid_clock.cc:648] HybridClock initialized: now 1781997362760792 us; error 51 us; skew 500 ppm
I20260620 23:16:02.761157 31520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:16:02 dist-test-slave-jkbq krb5kdc[31507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997362, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:02.764256 31520 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:16:02.765587 31520 webserver.cc:460] Webserver started at http://127.0.0.1:37559/ using document root <none> and password file <none>
I20260620 23:16:02.766192 31520 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:02.766288 31520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:02.766528 31520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:02.769483 31520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "95dbc356f3ff4382b4c21ed09b40ba20"
format_stamp: "Formatted at 2026-06-20 23:16:02 on dist-test-slave-jkbq"
server_key: "9579ec551ad6755702033f6216accd03"
server_key_iv: "7addf3516ca858259fba210b10374e07"
server_key_version: "encryptionkey@0"
I20260620 23:16:02.770212 31520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "95dbc356f3ff4382b4c21ed09b40ba20"
format_stamp: "Formatted at 2026-06-20 23:16:02 on dist-test-slave-jkbq"
server_key: "9579ec551ad6755702033f6216accd03"
server_key_iv: "7addf3516ca858259fba210b10374e07"
server_key_version: "encryptionkey@0"
I20260620 23:16:02.775375 31520 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 23:16:02.778633 31535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:02.779990 31520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:16:02.780174 31520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "95dbc356f3ff4382b4c21ed09b40ba20"
format_stamp: "Formatted at 2026-06-20 23:16:02 on dist-test-slave-jkbq"
server_key: "9579ec551ad6755702033f6216accd03"
server_key_iv: "7addf3516ca858259fba210b10374e07"
server_key_version: "encryptionkey@0"
I20260620 23:16:02.780382 31520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:02.804723 31520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:02.808816 31520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:02.809106 31520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:02.827847 31520 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42059
I20260620 23:16:02.827922 31586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42059 every 8 connection(s)
I20260620 23:16:02.829226 31520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:16:02.832839 31587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:02.834087 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 31520
I20260620 23:16:02.834286 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:16:02.834896 11007 external_mini_cluster.cc:1428] Setting key bf53c67f30fc5f7d282915483c86e729
I20260620 23:16:02.845407 31587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95dbc356f3ff4382b4c21ed09b40ba20: Bootstrap starting.
I20260620 23:16:02.847872 31590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:02.836769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57701 (local address 127.0.0.1:42059)
0620 23:16:02.837515 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:02.837527 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:02.837567 (+ 40us) server_negotiation.cc:407] Connection header received
0620 23:16:02.839338 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:02.839362 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:02.839754 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:02.840158 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:02.840675 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:02.841922 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:02.843325 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:02.843987 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:02.844179 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:02.845604 (+ 1425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:02.845612 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:02.845625 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:02.846148 (+ 523us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:02.846228 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:02.846252 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:02.846428 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:02.846746 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 23:16:02.846952 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":315,"threads_started":1}
I20260620 23:16:02.848682 31587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95dbc356f3ff4382b4c21ed09b40ba20: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:02.849720 31587 log.cc:824] T 00000000000000000000000000000000 P 95dbc356f3ff4382b4c21ed09b40ba20: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:02.851658 11007 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:16:02.852893 31587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95dbc356f3ff4382b4c21ed09b40ba20: No bootstrap required, opened a new log
I20260620 23:16:02.861351 31587 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95dbc356f3ff4382b4c21ed09b40ba20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95dbc356f3ff4382b4c21ed09b40ba20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42059 } }
I20260620 23:16:02.861759 31587 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95dbc356f3ff4382b4c21ed09b40ba20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:02.861879 31587 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95dbc356f3ff4382b4c21ed09b40ba20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95dbc356f3ff4382b4c21ed09b40ba20, State: Initialized, Role: FOLLOWER
I20260620 23:16:02.862169 31590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:02.855863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60347 (local address 127.0.0.1:42059)
0620 23:16:02.856086 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:02.856097 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:02.856435 (+ 338us) server_negotiation.cc:407] Connection header received
0620 23:16:02.856626 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:02.856632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:02.856702 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:02.856818 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:02.857321 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:02.858177 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:02.859532 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:02.859997 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:02.860143 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:02.861300 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:02.861306 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:02.861309 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:02.861326 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:02.861386 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:02.861404 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:02.861530 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:02.861729 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 23:16:02.861792 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 23:16:02.862574 31587 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95dbc356f3ff4382b4c21ed09b40ba20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95dbc356f3ff4382b4c21ed09b40ba20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42059 } }
I20260620 23:16:02.862835 31587 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95dbc356f3ff4382b4c21ed09b40ba20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:16:02.863029 31587 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95dbc356f3ff4382b4c21ed09b40ba20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:16:02.863215 31587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95dbc356f3ff4382b4c21ed09b40ba20 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:02.865563 31587 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95dbc356f3ff4382b4c21ed09b40ba20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95dbc356f3ff4382b4c21ed09b40ba20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42059 } }
I20260620 23:16:02.866070 31587 leader_election.cc:302] T 00000000000000000000000000000000 P 95dbc356f3ff4382b4c21ed09b40ba20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95dbc356f3ff4382b4c21ed09b40ba20; no voters:
I20260620 23:16:02.866806 31587 leader_election.cc:288] T 00000000000000000000000000000000 P 95dbc356f3ff4382b4c21ed09b40ba20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:16:02.866947 31597 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95dbc356f3ff4382b4c21ed09b40ba20 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:16:02.867615 31597 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95dbc356f3ff4382b4c21ed09b40ba20 [term 1 LEADER]: Becoming Leader. State: Replica: 95dbc356f3ff4382b4c21ed09b40ba20, State: Running, Role: LEADER
I20260620 23:16:02.868135 31597 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95dbc356f3ff4382b4c21ed09b40ba20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95dbc356f3ff4382b4c21ed09b40ba20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42059 } }
I20260620 23:16:02.869115 31587 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95dbc356f3ff4382b4c21ed09b40ba20 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:16:02.874701 31598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95dbc356f3ff4382b4c21ed09b40ba20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95dbc356f3ff4382b4c21ed09b40ba20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95dbc356f3ff4382b4c21ed09b40ba20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42059 } } }
I20260620 23:16:02.875155 31598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95dbc356f3ff4382b4c21ed09b40ba20 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:02.875120 31599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95dbc356f3ff4382b4c21ed09b40ba20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95dbc356f3ff4382b4c21ed09b40ba20. Latest consensus state: current_term: 1 leader_uuid: "95dbc356f3ff4382b4c21ed09b40ba20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95dbc356f3ff4382b4c21ed09b40ba20" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42059 } } }
I20260620 23:16:02.875489 31599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95dbc356f3ff4382b4c21ed09b40ba20 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:02.877379 31606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:16:02.882915 31606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:16:02.891350 31606 catalog_manager.cc:1347] Generated new cluster ID: 396f20952f7f492d84d6a82a30ff46ff
I20260620 23:16:02.891464 31606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:16:02.918020 31606 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:16:02.919850 31606 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:16:02.928817 31606 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95dbc356f3ff4382b4c21ed09b40ba20: Generated new TSK 0
I20260620 23:16:02.929811 31606 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:16:02.937879 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 31520
W20260620 23:16:02.938956 31582 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57701
W20260620 23:16:02.939733 11007 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 31520: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:16:02.939919 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 31520
W20260620 23:16:02.956606 31505 connection.cc:535] client connection to 127.0.0.1:42059 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42059): Connection reset by peer (error 104)
2026-06-20T23:16:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1876 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997042893763-11007-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:16:03 dist-test-slave-jkbq krb5kdc[31618](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:16:03 dist-test-slave-jkbq krb5kdc[31618](info): set up 2 sockets
Jun 20 23:16:03 dist-test-slave-jkbq krb5kdc[31618](info): commencing operation
krb5kdc: starting...
W20260620 23:16:04.043589 11007 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Jun 20 23:16:04 dist-test-slave-jkbq krb5kdc[31618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:16:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:16:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:16:04.290357 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37025
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:16:04.621868 31631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:04.622175 31631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:04.622282 31631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:04.629410 31631 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:16:04.629519 31631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:04.629563 31631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:04.629623 31631 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:16:04.629668 31631 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:16:04.639238 31631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37025
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:16:04.640406 31631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:04.641635 31631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:04.653086 31636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:04.652948 31637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:04.652923 31639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:04.654213 31631 server_base.cc:1034] running on GCE node
I20260620 23:16:04.655027 31631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:04.656455 31631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:04.657691 31631 hybrid_clock.cc:648] HybridClock initialized: now 1781997364657663 us; error 46 us; skew 500 ppm
I20260620 23:16:04.658028 31631 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:16:04 dist-test-slave-jkbq krb5kdc[31618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997364, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:04.666050 31631 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:16:04.667595 31631 webserver.cc:460] Webserver started at http://127.0.0.1:43535/ using document root <none> and password file <none>
I20260620 23:16:04.668233 31631 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:04.668344 31631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:04.668601 31631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:04.671551 31631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "668ba8dfc5064b1bba200f454d491233"
format_stamp: "Formatted at 2026-06-20 23:16:04 on dist-test-slave-jkbq"
server_key: "aace3c242307852da8b55517c308d612"
server_key_iv: "f1ea046dbf8208a63af61d8602ce3d46"
server_key_version: "encryptionkey@0"
I20260620 23:16:04.672246 31631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "668ba8dfc5064b1bba200f454d491233"
format_stamp: "Formatted at 2026-06-20 23:16:04 on dist-test-slave-jkbq"
server_key: "aace3c242307852da8b55517c308d612"
server_key_iv: "f1ea046dbf8208a63af61d8602ce3d46"
server_key_version: "encryptionkey@0"
I20260620 23:16:04.677246 31631 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:16:04.680619 31646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:04.681926 31631 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:16:04.682111 31631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "668ba8dfc5064b1bba200f454d491233"
format_stamp: "Formatted at 2026-06-20 23:16:04 on dist-test-slave-jkbq"
server_key: "aace3c242307852da8b55517c308d612"
server_key_iv: "f1ea046dbf8208a63af61d8602ce3d46"
server_key_version: "encryptionkey@0"
I20260620 23:16:04.682319 31631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:04.688133 31631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:04.692104 31631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:04.692399 31631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:04.714807 31631 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37025
I20260620 23:16:04.714843 31697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37025 every 8 connection(s)
I20260620 23:16:04.716267 31631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:16:04.719983 31698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:04.723605 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 31631
I20260620 23:16:04.723775 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:16:04.724313 11007 external_mini_cluster.cc:1428] Setting key 80e4160e092daf07829f7f3de922fc38
I20260620 23:16:04.731925 31701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:04.726089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57511 (local address 127.0.0.1:37025)
0620 23:16:04.726925 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:04.726942 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:04.726990 (+ 48us) server_negotiation.cc:407] Connection header received
0620 23:16:04.728606 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:04.728627 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:04.728973 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:04.729291 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:04.730375 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:04.730382 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:04.730398 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:04.730422 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:04.730491 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:04.730513 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:04.730690 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:04.730795 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 23:16:04.731035 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":352,"threads_started":1}
I20260620 23:16:04.732617 31698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 668ba8dfc5064b1bba200f454d491233: Bootstrap starting.
I20260620 23:16:04.735848 11007 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:16:04.736455 31698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 668ba8dfc5064b1bba200f454d491233: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:04.737673 31698 log.cc:824] T 00000000000000000000000000000000 P 668ba8dfc5064b1bba200f454d491233: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:04.741005 31698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 668ba8dfc5064b1bba200f454d491233: No bootstrap required, opened a new log
I20260620 23:16:04.743674 31701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:04.741022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52287 (local address 127.0.0.1:37025)
0620 23:16:04.741195 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:04.741204 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:04.741609 (+ 405us) server_negotiation.cc:407] Connection header received
0620 23:16:04.741810 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:04.741816 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:04.741871 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:04.741976 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:04.742962 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:04.742968 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:04.742972 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:04.742993 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:04.743039 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:04.743054 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:04.743200 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:04.743301 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 23:16:04.743364 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 23:16:04.749424 31698 raft_consensus.cc:348] T 00000000000000000000000000000000 P 668ba8dfc5064b1bba200f454d491233 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "668ba8dfc5064b1bba200f454d491233" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37025 } }
I20260620 23:16:04.749826 31698 raft_consensus.cc:374] T 00000000000000000000000000000000 P 668ba8dfc5064b1bba200f454d491233 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:04.749938 31698 raft_consensus.cc:729] T 00000000000000000000000000000000 P 668ba8dfc5064b1bba200f454d491233 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668ba8dfc5064b1bba200f454d491233, State: Initialized, Role: FOLLOWER
I20260620 23:16:04.750505 31698 consensus_queue.cc:260] T 00000000000000000000000000000000 P 668ba8dfc5064b1bba200f454d491233 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "668ba8dfc5064b1bba200f454d491233" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37025 } }
I20260620 23:16:04.750749 31698 raft_consensus.cc:388] T 00000000000000000000000000000000 P 668ba8dfc5064b1bba200f454d491233 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:16:04.750919 31698 raft_consensus.cc:482] T 00000000000000000000000000000000 P 668ba8dfc5064b1bba200f454d491233 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:16:04.751116 31698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 668ba8dfc5064b1bba200f454d491233 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:04.753363 31698 raft_consensus.cc:504] T 00000000000000000000000000000000 P 668ba8dfc5064b1bba200f454d491233 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "668ba8dfc5064b1bba200f454d491233" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37025 } }
I20260620 23:16:04.753983 31698 leader_election.cc:302] T 00000000000000000000000000000000 P 668ba8dfc5064b1bba200f454d491233 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 668ba8dfc5064b1bba200f454d491233; no voters:
I20260620 23:16:04.754670 31698 leader_election.cc:288] T 00000000000000000000000000000000 P 668ba8dfc5064b1bba200f454d491233 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:16:04.754770 31708 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 668ba8dfc5064b1bba200f454d491233 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:16:04.755481 31708 raft_consensus.cc:686] T 00000000000000000000000000000000 P 668ba8dfc5064b1bba200f454d491233 [term 1 LEADER]: Becoming Leader. State: Replica: 668ba8dfc5064b1bba200f454d491233, State: Running, Role: LEADER
I20260620 23:16:04.756042 31708 consensus_queue.cc:237] T 00000000000000000000000000000000 P 668ba8dfc5064b1bba200f454d491233 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "668ba8dfc5064b1bba200f454d491233" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37025 } }
I20260620 23:16:04.756765 31698 sys_catalog.cc:564] T 00000000000000000000000000000000 P 668ba8dfc5064b1bba200f454d491233 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:16:04.758806 31709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 668ba8dfc5064b1bba200f454d491233 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "668ba8dfc5064b1bba200f454d491233" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "668ba8dfc5064b1bba200f454d491233" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37025 } } }
I20260620 23:16:04.759289 31709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 668ba8dfc5064b1bba200f454d491233 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:04.759123 31710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 668ba8dfc5064b1bba200f454d491233 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 668ba8dfc5064b1bba200f454d491233. Latest consensus state: current_term: 1 leader_uuid: "668ba8dfc5064b1bba200f454d491233" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "668ba8dfc5064b1bba200f454d491233" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37025 } } }
I20260620 23:16:04.759527 31710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 668ba8dfc5064b1bba200f454d491233 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:04.760282 31716 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:16:04.765667 31716 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:16:04.774917 31716 catalog_manager.cc:1347] Generated new cluster ID: 3a087ba62c8d41d2af8d92df36bd8103
I20260620 23:16:04.775032 31716 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:16:04.794822 31716 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:16:04.795341 31716 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:16:04.812785 31716 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 668ba8dfc5064b1bba200f454d491233: Generated new TSK 0
I20260620 23:16:04.813854 31716 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:16:04.821575 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 31631
W20260620 23:16:04.822636 31693 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57511
W20260620 23:16:04.823343 11007 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 31631: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:16:04.823484 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 31631
2026-06-20T23:16:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1883 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997042893763-11007-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:16:04 dist-test-slave-jkbq krb5kdc[31729](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:16:04 dist-test-slave-jkbq krb5kdc[31729](info): set up 2 sockets
Jun 20 23:16:04 dist-test-slave-jkbq krb5kdc[31729](info): commencing operation
krb5kdc: starting...
W20260620 23:16:05.915215 11007 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.004s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Jun 20 23:16:06 dist-test-slave-jkbq krb5kdc[31729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:16:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:16:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:16:06.148294 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45417
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:16:06.474380 31742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:06.474736 31742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:06.474929 31742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:06.482241 31742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:16:06.482340 31742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:06.482431 31742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:06.482520 31742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:16:06.482583 31742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:16:06.492075 31742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45417
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:16:06.493316 31742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:06.494629 31742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:06.506072 31750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:06.506036 31748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:06.506175 31747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:06.507155 31742 server_base.cc:1034] running on GCE node
I20260620 23:16:06.507951 31742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:06.509351 31742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:06.510555 31742 hybrid_clock.cc:648] HybridClock initialized: now 1781997366510501 us; error 47 us; skew 500 ppm
I20260620 23:16:06.510957 31742 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:16:06 dist-test-slave-jkbq krb5kdc[31729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:06.514657 31742 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:16:06.516088 31742 webserver.cc:460] Webserver started at http://127.0.0.1:45753/ using document root <none> and password file <none>
I20260620 23:16:06.516737 31742 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:06.516870 31742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:06.517181 31742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:06.520210 31742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "932a2a28c8f24e07bc63fc5cbfad8953"
format_stamp: "Formatted at 2026-06-20 23:16:06 on dist-test-slave-jkbq"
server_key: "b4f96e7c2c79e0e007bbdd20bbc93431"
server_key_iv: "a15f7dfb148b038b2b92a2aa9678ec4d"
server_key_version: "encryptionkey@0"
I20260620 23:16:06.520953 31742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "932a2a28c8f24e07bc63fc5cbfad8953"
format_stamp: "Formatted at 2026-06-20 23:16:06 on dist-test-slave-jkbq"
server_key: "b4f96e7c2c79e0e007bbdd20bbc93431"
server_key_iv: "a15f7dfb148b038b2b92a2aa9678ec4d"
server_key_version: "encryptionkey@0"
I20260620 23:16:06.525831 31742 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 23:16:06.529150 31757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:06.530490 31742 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 23:16:06.530676 31742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "932a2a28c8f24e07bc63fc5cbfad8953"
format_stamp: "Formatted at 2026-06-20 23:16:06 on dist-test-slave-jkbq"
server_key: "b4f96e7c2c79e0e007bbdd20bbc93431"
server_key_iv: "a15f7dfb148b038b2b92a2aa9678ec4d"
server_key_version: "encryptionkey@0"
I20260620 23:16:06.530864 31742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:06.547232 31742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:06.551309 31742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:06.551584 31742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:06.569984 31742 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45417
I20260620 23:16:06.570016 31808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45417 every 8 connection(s)
I20260620 23:16:06.571385 31742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:16:06.575179 31809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:06.580900 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 31742
I20260620 23:16:06.581099 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:16:06.581524 11007 external_mini_cluster.cc:1428] Setting key 9ed344560653caca2d91f70a91e31e1b
I20260620 23:16:06.585661 31809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 932a2a28c8f24e07bc63fc5cbfad8953: Bootstrap starting.
I20260620 23:16:06.589227 31809 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 932a2a28c8f24e07bc63fc5cbfad8953: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:06.590451 31809 log.cc:824] T 00000000000000000000000000000000 P 932a2a28c8f24e07bc63fc5cbfad8953: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:06.593492 31809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 932a2a28c8f24e07bc63fc5cbfad8953: No bootstrap required, opened a new log
Jun 20 23:16:06 dist-test-slave-jkbq krb5kdc[31729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 23:16:06.601771 31809 raft_consensus.cc:348] T 00000000000000000000000000000000 P 932a2a28c8f24e07bc63fc5cbfad8953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932a2a28c8f24e07bc63fc5cbfad8953" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45417 } }
I20260620 23:16:06.602156 31809 raft_consensus.cc:374] T 00000000000000000000000000000000 P 932a2a28c8f24e07bc63fc5cbfad8953 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:06.602289 31809 raft_consensus.cc:729] T 00000000000000000000000000000000 P 932a2a28c8f24e07bc63fc5cbfad8953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 932a2a28c8f24e07bc63fc5cbfad8953, State: Initialized, Role: FOLLOWER
I20260620 23:16:06.602996 31809 consensus_queue.cc:260] T 00000000000000000000000000000000 P 932a2a28c8f24e07bc63fc5cbfad8953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932a2a28c8f24e07bc63fc5cbfad8953" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45417 } }
I20260620 23:16:06.603295 31809 raft_consensus.cc:388] T 00000000000000000000000000000000 P 932a2a28c8f24e07bc63fc5cbfad8953 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:16:06.603451 31809 raft_consensus.cc:482] T 00000000000000000000000000000000 P 932a2a28c8f24e07bc63fc5cbfad8953 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:16:06.603650 31809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 932a2a28c8f24e07bc63fc5cbfad8953 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:06.604275 31812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:06.583267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33415 (local address 127.0.0.1:45417)
0620 23:16:06.584078 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:06.584089 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:06.584127 (+ 38us) server_negotiation.cc:407] Connection header received
0620 23:16:06.585695 (+ 1568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:06.585716 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:06.586151 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:06.586454 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:06.588211 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:06.589513 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:06.591306 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:06.592113 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:06.592239 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:06.596991 (+ 4752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:06.596999 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:06.597011 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:06.597047 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:06.600152 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:06.600861 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:06.600867 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:06.600873 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:06.601032 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:06.601444 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:06.601454 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:06.601456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:06.601947 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:06.602280 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:06.602711 (+ 431us) server_negotiation.cc:299] Negotiation successful
0620 23:16:06.603154 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":388,"threads_started":1}
I20260620 23:16:06.606477 31809 raft_consensus.cc:504] T 00000000000000000000000000000000 P 932a2a28c8f24e07bc63fc5cbfad8953 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932a2a28c8f24e07bc63fc5cbfad8953" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45417 } }
I20260620 23:16:06.607033 31809 leader_election.cc:302] T 00000000000000000000000000000000 P 932a2a28c8f24e07bc63fc5cbfad8953 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 932a2a28c8f24e07bc63fc5cbfad8953; no voters:
I20260620 23:16:06.608059 31809 leader_election.cc:288] T 00000000000000000000000000000000 P 932a2a28c8f24e07bc63fc5cbfad8953 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:16:06.608098 11007 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:16:06.608189 31814 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 932a2a28c8f24e07bc63fc5cbfad8953 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:16:06.609418 31814 raft_consensus.cc:686] T 00000000000000000000000000000000 P 932a2a28c8f24e07bc63fc5cbfad8953 [term 1 LEADER]: Becoming Leader. State: Replica: 932a2a28c8f24e07bc63fc5cbfad8953, State: Running, Role: LEADER
I20260620 23:16:06.610131 31809 sys_catalog.cc:564] T 00000000000000000000000000000000 P 932a2a28c8f24e07bc63fc5cbfad8953 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:16:06.610005 31814 consensus_queue.cc:237] T 00000000000000000000000000000000 P 932a2a28c8f24e07bc63fc5cbfad8953 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932a2a28c8f24e07bc63fc5cbfad8953" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45417 } }
I20260620 23:16:06.616990 31815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 932a2a28c8f24e07bc63fc5cbfad8953 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "932a2a28c8f24e07bc63fc5cbfad8953" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932a2a28c8f24e07bc63fc5cbfad8953" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45417 } } }
I20260620 23:16:06.617430 31815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 932a2a28c8f24e07bc63fc5cbfad8953 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:06.618029 31816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 932a2a28c8f24e07bc63fc5cbfad8953 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 932a2a28c8f24e07bc63fc5cbfad8953. Latest consensus state: current_term: 1 leader_uuid: "932a2a28c8f24e07bc63fc5cbfad8953" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932a2a28c8f24e07bc63fc5cbfad8953" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45417 } } }
I20260620 23:16:06.618360 31816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 932a2a28c8f24e07bc63fc5cbfad8953 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:06.624907 31825 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:16:06.630208 31825 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:16:06.636400 31812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:06.613269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41431 (local address 127.0.0.1:45417)
0620 23:16:06.613488 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:06.613496 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:06.615084 (+ 1588us) server_negotiation.cc:407] Connection header received
0620 23:16:06.615280 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:06.615288 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:06.615380 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:06.615488 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:06.617587 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:06.619095 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:06.621606 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:06.622580 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:06.622704 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:06.626206 (+ 3502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:06.626215 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:06.626223 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:06.626272 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:06.631261 (+ 4989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:06.632336 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:06.632344 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:06.632350 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:06.632525 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:06.633081 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:06.633088 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:06.633094 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:06.633726 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:06.635405 (+ 1679us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:06.635585 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 23:16:06.635872 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 23:16:06.645419 31825 catalog_manager.cc:1347] Generated new cluster ID: 55f391678042493eac9889cfe1b3e5bb
I20260620 23:16:06.645524 31825 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:16:06.656522 31825 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:16:06.657724 31825 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:16:06.673138 31825 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 932a2a28c8f24e07bc63fc5cbfad8953: Generated new TSK 0
I20260620 23:16:06.673925 31825 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:16:06.687364 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 31742
I20260620 23:16:06.688376 31804 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:16:06.785162 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 31742
2026-06-20T23:16:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1974 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997042893763-11007-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:16:06 dist-test-slave-jkbq krb5kdc[31842](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:16:06 dist-test-slave-jkbq krb5kdc[31842](info): set up 2 sockets
Jun 20 23:16:06 dist-test-slave-jkbq krb5kdc[31842](info): commencing operation
krb5kdc: starting...
W20260620 23:16:07.899258 11007 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Jun 20 23:16:08 dist-test-slave-jkbq krb5kdc[31842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:16:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:16:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:16:08.144433 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38891
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:16:08.485715 31855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:08.485997 31855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:08.486089 31855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:08.493402 31855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:16:08.493518 31855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:08.493573 31855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:08.493633 31855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:16:08.493675 31855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:16:08.503615 31855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:16:08.504827 31855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:08.506069 31855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:08.517869 31860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:08.517879 31861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:08.517930 31863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:08.518797 31855 server_base.cc:1034] running on GCE node
I20260620 23:16:08.519613 31855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:08.520951 31855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:08.522164 31855 hybrid_clock.cc:648] HybridClock initialized: now 1781997368522127 us; error 28 us; skew 500 ppm
I20260620 23:16:08.522523 31855 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:16:08 dist-test-slave-jkbq krb5kdc[31842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:08.525707 31855 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:16:08.527266 31855 webserver.cc:460] Webserver started at http://127.0.0.1:39797/ using document root <none> and password file <none>
I20260620 23:16:08.527940 31855 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:08.528088 31855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:08.528404 31855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:08.531445 31855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "9c915c4b3aa24a5a823c25b45f910cd3"
format_stamp: "Formatted at 2026-06-20 23:16:08 on dist-test-slave-jkbq"
server_key: "31a6fab9a41a55c2c698edbcea8399a6"
server_key_iv: "bd1de2b17c0ae2c3025f54d794d29938"
server_key_version: "encryptionkey@0"
I20260620 23:16:08.532227 31855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "9c915c4b3aa24a5a823c25b45f910cd3"
format_stamp: "Formatted at 2026-06-20 23:16:08 on dist-test-slave-jkbq"
server_key: "31a6fab9a41a55c2c698edbcea8399a6"
server_key_iv: "bd1de2b17c0ae2c3025f54d794d29938"
server_key_version: "encryptionkey@0"
I20260620 23:16:08.537293 31855 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:16:08.540748 31870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:08.542037 31855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:16:08.542243 31855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "9c915c4b3aa24a5a823c25b45f910cd3"
format_stamp: "Formatted at 2026-06-20 23:16:08 on dist-test-slave-jkbq"
server_key: "31a6fab9a41a55c2c698edbcea8399a6"
server_key_iv: "bd1de2b17c0ae2c3025f54d794d29938"
server_key_version: "encryptionkey@0"
I20260620 23:16:08.542450 31855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:08.574090 31855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:08.578097 31855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:08.578401 31855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:08.597186 31855 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38891
I20260620 23:16:08.597225 31921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38891 every 8 connection(s)
I20260620 23:16:08.598601 31855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:16:08.602362 31922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:08.608417 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 31855
I20260620 23:16:08.608615 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:16:08.609145 11007 external_mini_cluster.cc:1428] Setting key 1b8cd0938e307fe8ecb2c796c0a9b38c
I20260620 23:16:08.613916 31922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c915c4b3aa24a5a823c25b45f910cd3: Bootstrap starting.
I20260620 23:16:08.617012 31922 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c915c4b3aa24a5a823c25b45f910cd3: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:08.618013 31922 log.cc:824] T 00000000000000000000000000000000 P 9c915c4b3aa24a5a823c25b45f910cd3: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:08.620818 31922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c915c4b3aa24a5a823c25b45f910cd3: No bootstrap required, opened a new log
I20260620 23:16:08.622208 31925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:08.611075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58943 (local address 127.0.0.1:38891)
0620 23:16:08.611952 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:08.611965 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:08.612008 (+ 43us) server_negotiation.cc:407] Connection header received
0620 23:16:08.613628 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:08.613651 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:08.614099 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:08.614511 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:08.615064 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:08.616395 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:08.617817 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:08.618576 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:08.618788 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:08.619972 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:08.619980 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:08.619997 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:08.620521 (+ 524us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:08.620596 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:08.620621 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:08.620757 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:08.620984 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 23:16:08.621266 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":346,"threads_started":1}
I20260620 23:16:08.626080 11007 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:16:08.631057 31922 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c915c4b3aa24a5a823c25b45f910cd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c915c4b3aa24a5a823c25b45f910cd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38891 } }
I20260620 23:16:08.631670 31922 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c915c4b3aa24a5a823c25b45f910cd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:08.631788 31922 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c915c4b3aa24a5a823c25b45f910cd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c915c4b3aa24a5a823c25b45f910cd3, State: Initialized, Role: FOLLOWER
I20260620 23:16:08.632480 31922 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c915c4b3aa24a5a823c25b45f910cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c915c4b3aa24a5a823c25b45f910cd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38891 } }
I20260620 23:16:08.632808 31922 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c915c4b3aa24a5a823c25b45f910cd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:16:08.632970 31922 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c915c4b3aa24a5a823c25b45f910cd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:16:08.633172 31922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c915c4b3aa24a5a823c25b45f910cd3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:08.636304 31922 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c915c4b3aa24a5a823c25b45f910cd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c915c4b3aa24a5a823c25b45f910cd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38891 } }
I20260620 23:16:08.636940 31922 leader_election.cc:302] T 00000000000000000000000000000000 P 9c915c4b3aa24a5a823c25b45f910cd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c915c4b3aa24a5a823c25b45f910cd3; no voters:
I20260620 23:16:08.637665 31922 leader_election.cc:288] T 00000000000000000000000000000000 P 9c915c4b3aa24a5a823c25b45f910cd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:16:08.637778 31932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c915c4b3aa24a5a823c25b45f910cd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:16:08.638244 31925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:08.631282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43883 (local address 127.0.0.1:38891)
0620 23:16:08.631506 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:08.631512 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:08.631846 (+ 334us) server_negotiation.cc:407] Connection header received
0620 23:16:08.632045 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:08.632065 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:08.632153 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:08.632262 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:08.632933 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:08.633803 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:08.635678 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:08.636103 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:08.636196 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:08.637346 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:08.637370 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:08.637376 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:08.637401 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:08.637459 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:08.637485 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:08.637598 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:08.637752 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 23:16:08.637869 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:16:08.639149 31932 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c915c4b3aa24a5a823c25b45f910cd3 [term 1 LEADER]: Becoming Leader. State: Replica: 9c915c4b3aa24a5a823c25b45f910cd3, State: Running, Role: LEADER
I20260620 23:16:08.639791 31932 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c915c4b3aa24a5a823c25b45f910cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c915c4b3aa24a5a823c25b45f910cd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38891 } }
I20260620 23:16:08.640074 31922 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c915c4b3aa24a5a823c25b45f910cd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:16:08.642238 31934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c915c4b3aa24a5a823c25b45f910cd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c915c4b3aa24a5a823c25b45f910cd3. Latest consensus state: current_term: 1 leader_uuid: "9c915c4b3aa24a5a823c25b45f910cd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c915c4b3aa24a5a823c25b45f910cd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38891 } } }
I20260620 23:16:08.642576 31934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c915c4b3aa24a5a823c25b45f910cd3 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:08.643193 31933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c915c4b3aa24a5a823c25b45f910cd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c915c4b3aa24a5a823c25b45f910cd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c915c4b3aa24a5a823c25b45f910cd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38891 } } }
I20260620 23:16:08.643527 31933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c915c4b3aa24a5a823c25b45f910cd3 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:08.650899 31939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:16:08.655267 31939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:16:08.663973 31939 catalog_manager.cc:1347] Generated new cluster ID: e85057930d4f4800b5c483b340c0daa7
I20260620 23:16:08.664090 31939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:16:08.706935 31939 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:16:08.708329 31939 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:16:08.715509 31939 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c915c4b3aa24a5a823c25b45f910cd3: Generated new TSK 0
I20260620 23:16:08.716346 31939 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:16:08.724526 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 31855
W20260620 23:16:08.725631 31917 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58943
W20260620 23:16:08.726358 11007 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 31855: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:16:08.726503 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 31855
2026-06-20T23:16:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1925 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997042893763-11007-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:16:08 dist-test-slave-jkbq krb5kdc[31953](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:16:08 dist-test-slave-jkbq krb5kdc[31953](info): set up 2 sockets
Jun 20 23:16:08 dist-test-slave-jkbq krb5kdc[31953](info): commencing operation
krb5kdc: starting...
W20260620 23:16:09.827319 11007 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.010s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Jun 20 23:16:09 dist-test-slave-jkbq krb5kdc[31953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:16:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:16:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:16:10.073956 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32913
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:16:10.398280 31966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:10.398638 31966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:10.398797 31966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:10.405977 31966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:16:10.406129 31966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:10.406239 31966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:10.406358 31966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:16:10.406446 31966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:16:10.416219 31966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:16:10.417608 31966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:10.419114 31966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:10.430588 31974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:10.430619 31972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:10.431054 31971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:10.431408 31966 server_base.cc:1034] running on GCE node
I20260620 23:16:10.432133 31966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:10.433491 31966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:10.434736 31966 hybrid_clock.cc:648] HybridClock initialized: now 1781997370434702 us; error 48 us; skew 500 ppm
I20260620 23:16:10.435132 31966 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:16:10 dist-test-slave-jkbq krb5kdc[31953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997370, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:10.442591 31966 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:16:10.444216 31966 webserver.cc:460] Webserver started at http://127.0.0.1:37693/ using document root <none> and password file <none>
I20260620 23:16:10.444875 31966 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:10.444980 31966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:10.445232 31966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:10.448276 31966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "9560fae206e34d1c8ff6517cffb78e60"
format_stamp: "Formatted at 2026-06-20 23:16:10 on dist-test-slave-jkbq"
server_key: "25ded807414ddebac8bf1b8961069b2e"
server_key_iv: "3d0699fa39a815cf4b686255d45fb956"
server_key_version: "encryptionkey@0"
I20260620 23:16:10.448982 31966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "9560fae206e34d1c8ff6517cffb78e60"
format_stamp: "Formatted at 2026-06-20 23:16:10 on dist-test-slave-jkbq"
server_key: "25ded807414ddebac8bf1b8961069b2e"
server_key_iv: "3d0699fa39a815cf4b686255d45fb956"
server_key_version: "encryptionkey@0"
I20260620 23:16:10.454140 31966 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260620 23:16:10.457585 31981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:10.458985 31966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 23:16:10.459162 31966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "9560fae206e34d1c8ff6517cffb78e60"
format_stamp: "Formatted at 2026-06-20 23:16:10 on dist-test-slave-jkbq"
server_key: "25ded807414ddebac8bf1b8961069b2e"
server_key_iv: "3d0699fa39a815cf4b686255d45fb956"
server_key_version: "encryptionkey@0"
I20260620 23:16:10.459344 31966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:10.464897 31966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:10.469189 31966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:10.469502 31966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:10.488178 31966 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32913
I20260620 23:16:10.488243 32032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32913 every 8 connection(s)
I20260620 23:16:10.490132 31966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:16:10.493958 32033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:10.496572 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 31966
I20260620 23:16:10.496738 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:16:10.497246 11007 external_mini_cluster.cc:1428] Setting key 0ff4f22d6b67f490e29531a34b2cb104
I20260620 23:16:10.504916 32036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:10.499128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33079 (local address 127.0.0.1:32913)
0620 23:16:10.499878 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:10.499893 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:10.499937 (+ 44us) server_negotiation.cc:407] Connection header received
0620 23:16:10.501535 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:10.501555 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:10.501878 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:10.502275 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:10.503368 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:10.503375 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:10.503390 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:10.503402 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:10.503453 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:10.503474 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:10.503729 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:10.503869 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 23:16:10.504043 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":333,"threads_started":1}
I20260620 23:16:10.506034 32033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9560fae206e34d1c8ff6517cffb78e60: Bootstrap starting.
I20260620 23:16:10.508504 11007 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:16:10.509375 32033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9560fae206e34d1c8ff6517cffb78e60: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:10.510371 32033 log.cc:824] T 00000000000000000000000000000000 P 9560fae206e34d1c8ff6517cffb78e60: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:10.513353 32033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9560fae206e34d1c8ff6517cffb78e60: No bootstrap required, opened a new log
I20260620 23:16:10.515427 32036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:10.512911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53975 (local address 127.0.0.1:32913)
0620 23:16:10.513092 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:10.513099 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:10.513614 (+ 515us) server_negotiation.cc:407] Connection header received
0620 23:16:10.513661 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:10.513664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:10.513719 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:10.513832 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:10.514741 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:10.514745 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:10.514748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:10.514762 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:10.514797 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:10.514812 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:10.514977 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:10.515099 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 23:16:10.515161 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:16:10.521538 32033 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9560fae206e34d1c8ff6517cffb78e60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9560fae206e34d1c8ff6517cffb78e60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32913 } }
I20260620 23:16:10.521915 32033 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9560fae206e34d1c8ff6517cffb78e60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:10.522011 32033 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9560fae206e34d1c8ff6517cffb78e60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9560fae206e34d1c8ff6517cffb78e60, State: Initialized, Role: FOLLOWER
I20260620 23:16:10.522701 32033 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9560fae206e34d1c8ff6517cffb78e60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9560fae206e34d1c8ff6517cffb78e60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32913 } }
I20260620 23:16:10.523056 32033 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9560fae206e34d1c8ff6517cffb78e60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:16:10.523218 32033 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9560fae206e34d1c8ff6517cffb78e60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:16:10.523406 32033 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9560fae206e34d1c8ff6517cffb78e60 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:10.525537 32033 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9560fae206e34d1c8ff6517cffb78e60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9560fae206e34d1c8ff6517cffb78e60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32913 } }
I20260620 23:16:10.526026 32033 leader_election.cc:302] T 00000000000000000000000000000000 P 9560fae206e34d1c8ff6517cffb78e60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9560fae206e34d1c8ff6517cffb78e60; no voters:
I20260620 23:16:10.526630 32033 leader_election.cc:288] T 00000000000000000000000000000000 P 9560fae206e34d1c8ff6517cffb78e60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:16:10.526791 32043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9560fae206e34d1c8ff6517cffb78e60 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:16:10.527586 32043 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9560fae206e34d1c8ff6517cffb78e60 [term 1 LEADER]: Becoming Leader. State: Replica: 9560fae206e34d1c8ff6517cffb78e60, State: Running, Role: LEADER
I20260620 23:16:10.528119 32043 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9560fae206e34d1c8ff6517cffb78e60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9560fae206e34d1c8ff6517cffb78e60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32913 } }
I20260620 23:16:10.528867 32033 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9560fae206e34d1c8ff6517cffb78e60 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:16:10.530834 32045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9560fae206e34d1c8ff6517cffb78e60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9560fae206e34d1c8ff6517cffb78e60. Latest consensus state: current_term: 1 leader_uuid: "9560fae206e34d1c8ff6517cffb78e60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9560fae206e34d1c8ff6517cffb78e60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32913 } } }
I20260620 23:16:10.531240 32044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9560fae206e34d1c8ff6517cffb78e60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9560fae206e34d1c8ff6517cffb78e60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9560fae206e34d1c8ff6517cffb78e60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32913 } } }
I20260620 23:16:10.532378 32045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9560fae206e34d1c8ff6517cffb78e60 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:10.532439 32044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9560fae206e34d1c8ff6517cffb78e60 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:10.534179 32051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:16:10.539268 32051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:16:10.549170 32051 catalog_manager.cc:1347] Generated new cluster ID: 43f22c163cc44a62ae5261054e39a095
I20260620 23:16:10.549279 32051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:16:10.578521 32051 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:16:10.579025 32051 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:16:10.586794 32051 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9560fae206e34d1c8ff6517cffb78e60: Generated new TSK 0
I20260620 23:16:10.587581 32051 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:16:10.599669 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 31966
W20260620 23:16:10.600718 32028 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33079
W20260620 23:16:10.601408 11007 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 31966: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:16:10.601542 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 31966
2026-06-20T23:16:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1877 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997042893763-11007-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:16:10 dist-test-slave-jkbq krb5kdc[32064](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:16:10 dist-test-slave-jkbq krb5kdc[32064](info): set up 2 sockets
Jun 20 23:16:10 dist-test-slave-jkbq krb5kdc[32064](info): commencing operation
krb5kdc: starting...
W20260620 23:16:11.711251 11007 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.007s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Jun 20 23:16:11 dist-test-slave-jkbq krb5kdc[32064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:16:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:16:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:16:11.957602 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34499
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:16:12.278748 32077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:12.279049 32077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:12.279211 32077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:12.290409 32077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:16:12.290529 32077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:12.290629 32077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:12.290731 32077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:16:12.290794 32077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:16:12.300928 32077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:16:12.302255 32077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:12.303654 32077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:12.315395 32082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:12.315383 32083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:12.315562 32085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:12.316275 32077 server_base.cc:1034] running on GCE node
I20260620 23:16:12.316962 32077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:12.318234 32077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:12.319440 32077 hybrid_clock.cc:648] HybridClock initialized: now 1781997372319388 us; error 43 us; skew 500 ppm
I20260620 23:16:12.319799 32077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:16:12 dist-test-slave-jkbq krb5kdc[32064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:12.322963 32077 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:16:12.324345 32077 webserver.cc:460] Webserver started at http://127.0.0.1:45041/ using document root <none> and password file <none>
I20260620 23:16:12.324992 32077 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:12.325124 32077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:12.325428 32077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:12.328374 32077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "83b8fc6a85fa441fade9562ed80ba632"
format_stamp: "Formatted at 2026-06-20 23:16:12 on dist-test-slave-jkbq"
server_key: "1827c7b74d12d04c4ef9f3cbad9aff91"
server_key_iv: "cdb03e826c873785f3068d40bc39f9a2"
server_key_version: "encryptionkey@0"
I20260620 23:16:12.329141 32077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "83b8fc6a85fa441fade9562ed80ba632"
format_stamp: "Formatted at 2026-06-20 23:16:12 on dist-test-slave-jkbq"
server_key: "1827c7b74d12d04c4ef9f3cbad9aff91"
server_key_iv: "cdb03e826c873785f3068d40bc39f9a2"
server_key_version: "encryptionkey@0"
I20260620 23:16:12.334147 32077 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 23:16:12.337595 32092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:12.338984 32077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 23:16:12.339161 32077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "83b8fc6a85fa441fade9562ed80ba632"
format_stamp: "Formatted at 2026-06-20 23:16:12 on dist-test-slave-jkbq"
server_key: "1827c7b74d12d04c4ef9f3cbad9aff91"
server_key_iv: "cdb03e826c873785f3068d40bc39f9a2"
server_key_version: "encryptionkey@0"
I20260620 23:16:12.339329 32077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:12.356599 32077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:12.360814 32077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:12.361097 32077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:12.379400 32077 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34499
I20260620 23:16:12.379431 32143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34499 every 8 connection(s)
I20260620 23:16:12.380874 32077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:16:12.384523 32144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:12.390055 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 32077
I20260620 23:16:12.390246 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:16:12.390836 11007 external_mini_cluster.cc:1428] Setting key 320ded9d6738fa6664d3d9e187b0d5bb
I20260620 23:16:12.395668 32144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83b8fc6a85fa441fade9562ed80ba632: Bootstrap starting.
I20260620 23:16:12.399474 32144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83b8fc6a85fa441fade9562ed80ba632: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:12.400738 32144 log.cc:824] T 00000000000000000000000000000000 P 83b8fc6a85fa441fade9562ed80ba632: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:12.403640 32144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83b8fc6a85fa441fade9562ed80ba632: No bootstrap required, opened a new log
Jun 20 23:16:12 dist-test-slave-jkbq krb5kdc[32064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 23:16:12.414119 32144 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83b8fc6a85fa441fade9562ed80ba632 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b8fc6a85fa441fade9562ed80ba632" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34499 } }
I20260620 23:16:12.414589 32144 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83b8fc6a85fa441fade9562ed80ba632 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:12.414716 32144 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83b8fc6a85fa441fade9562ed80ba632 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b8fc6a85fa441fade9562ed80ba632, State: Initialized, Role: FOLLOWER
I20260620 23:16:12.415573 32147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:12.392685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.227:32889 (local address 127.0.0.1:34499)
0620 23:16:12.393539 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:12.393551 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:12.393596 (+ 45us) server_negotiation.cc:407] Connection header received
0620 23:16:12.395341 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:12.395364 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:12.395766 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:12.396123 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:12.397950 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:12.399331 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:12.401152 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:12.401901 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:12.402132 (+ 231us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:12.406467 (+ 4335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:12.406476 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:12.406500 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:12.406555 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:12.410535 (+ 3980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:12.411494 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:12.411503 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:12.411512 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:12.411725 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:12.412252 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:12.412261 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:12.412265 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:12.413004 (+ 739us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:12.413317 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:12.413761 (+ 444us) server_negotiation.cc:299] Negotiation successful
0620 23:16:12.414108 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":414,"threads_started":1}
I20260620 23:16:12.415427 32144 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83b8fc6a85fa441fade9562ed80ba632 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b8fc6a85fa441fade9562ed80ba632" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34499 } }
I20260620 23:16:12.415722 32144 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83b8fc6a85fa441fade9562ed80ba632 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:16:12.415868 32144 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83b8fc6a85fa441fade9562ed80ba632 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:16:12.416049 32144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83b8fc6a85fa441fade9562ed80ba632 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:12.418524 32144 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83b8fc6a85fa441fade9562ed80ba632 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b8fc6a85fa441fade9562ed80ba632" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34499 } }
I20260620 23:16:12.419107 32144 leader_election.cc:302] T 00000000000000000000000000000000 P 83b8fc6a85fa441fade9562ed80ba632 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83b8fc6a85fa441fade9562ed80ba632; no voters:
I20260620 23:16:12.419538 11007 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:16:12.419752 32144 leader_election.cc:288] T 00000000000000000000000000000000 P 83b8fc6a85fa441fade9562ed80ba632 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:16:12.419894 32149 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83b8fc6a85fa441fade9562ed80ba632 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:16:12.420507 32149 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83b8fc6a85fa441fade9562ed80ba632 [term 1 LEADER]: Becoming Leader. State: Replica: 83b8fc6a85fa441fade9562ed80ba632, State: Running, Role: LEADER
I20260620 23:16:12.421053 32149 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83b8fc6a85fa441fade9562ed80ba632 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b8fc6a85fa441fade9562ed80ba632" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34499 } }
I20260620 23:16:12.423079 32149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83b8fc6a85fa441fade9562ed80ba632 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83b8fc6a85fa441fade9562ed80ba632. Latest consensus state: current_term: 1 leader_uuid: "83b8fc6a85fa441fade9562ed80ba632" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b8fc6a85fa441fade9562ed80ba632" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34499 } } }
I20260620 23:16:12.423367 32149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83b8fc6a85fa441fade9562ed80ba632 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:12.423532 32151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83b8fc6a85fa441fade9562ed80ba632 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83b8fc6a85fa441fade9562ed80ba632" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b8fc6a85fa441fade9562ed80ba632" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34499 } } }
I20260620 23:16:12.423913 32151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83b8fc6a85fa441fade9562ed80ba632 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:12.428421 32144 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83b8fc6a85fa441fade9562ed80ba632 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:16:12.435432 32153 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:16:12.444046 32153 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:16:12.457036 32153 catalog_manager.cc:1347] Generated new cluster ID: 78985a2d32fc47f5a917bde1131968c3
I20260620 23:16:12.457152 32153 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:16:12.461601 32147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:12.438902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.227:48451 (local address 127.0.0.1:34499)
0620 23:16:12.439114 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:12.439124 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:12.439157 (+ 33us) server_negotiation.cc:407] Connection header received
0620 23:16:12.439245 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:12.439252 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:12.439375 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:12.439522 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:12.441488 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:12.442701 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:12.444769 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:12.445462 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:12.445576 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:12.452247 (+ 6671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:12.452254 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:12.452258 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:12.452298 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:12.456753 (+ 4455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:12.458086 (+ 1333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:12.458094 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:12.458101 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:12.458288 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:12.459086 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:12.459093 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:12.459099 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:12.459624 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:12.459940 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:12.460804 (+ 864us) server_negotiation.cc:299] Negotiation successful
0620 23:16:12.461060 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 23:16:12.485814 32153 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:16:12.487233 32153 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:16:12.495374 32153 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83b8fc6a85fa441fade9562ed80ba632: Generated new TSK 0
I20260620 23:16:12.496433 32153 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:16:12.504976 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 32077
I20260620 23:16:12.506326 32139 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:16:12.598598 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 32077
W20260620 23:16:12.615590 32062 connection.cc:535] client connection to 127.0.0.1:34499 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34499): Connection reset by peer (error 104)
2026-06-20T23:16:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1997 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997042893763-11007-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:16:12 dist-test-slave-jkbq krb5kdc[32176](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:16:12 dist-test-slave-jkbq krb5kdc[32176](info): set up 2 sockets
Jun 20 23:16:12 dist-test-slave-jkbq krb5kdc[32176](info): commencing operation
krb5kdc: starting...
W20260620 23:16:13.707324 11007 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Jun 20 23:16:13 dist-test-slave-jkbq krb5kdc[32176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:16:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:16:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:16:13.957645 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42303
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:16:14.283764 32189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:14.284211 32189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:14.284372 32189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:14.292634 32189 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:16:14.292737 32189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:14.292784 32189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:14.292845 32189 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:16:14.292889 32189 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:16:14.302515 32189 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42303
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:16:14.303777 32189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:14.305027 32189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:14.316608 32194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:14.316608 32195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:14.316916 32197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:14.317330 32189 server_base.cc:1034] running on GCE node
I20260620 23:16:14.318069 32189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:14.319489 32189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:14.320688 32189 hybrid_clock.cc:648] HybridClock initialized: now 1781997374320641 us; error 29 us; skew 500 ppm
I20260620 23:16:14.321027 32189 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:16:14 dist-test-slave-jkbq krb5kdc[32176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997374, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:14.324106 32189 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:16:14.325551 32189 webserver.cc:460] Webserver started at http://127.0.0.1:46851/ using document root <none> and password file <none>
I20260620 23:16:14.326278 32189 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:14.326421 32189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:14.326730 32189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:14.329715 32189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "1d49c484f2b946ceb77b81635ab759e2"
format_stamp: "Formatted at 2026-06-20 23:16:14 on dist-test-slave-jkbq"
server_key: "5e75de39c5b86b8b70c7a37a3833deeb"
server_key_iv: "ed4b27d734aef96f0ab8e1f999cd5177"
server_key_version: "encryptionkey@0"
I20260620 23:16:14.330471 32189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "1d49c484f2b946ceb77b81635ab759e2"
format_stamp: "Formatted at 2026-06-20 23:16:14 on dist-test-slave-jkbq"
server_key: "5e75de39c5b86b8b70c7a37a3833deeb"
server_key_iv: "ed4b27d734aef96f0ab8e1f999cd5177"
server_key_version: "encryptionkey@0"
I20260620 23:16:14.335520 32189 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 23:16:14.338789 32204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:14.340168 32189 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:16:14.340389 32189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "1d49c484f2b946ceb77b81635ab759e2"
format_stamp: "Formatted at 2026-06-20 23:16:14 on dist-test-slave-jkbq"
server_key: "5e75de39c5b86b8b70c7a37a3833deeb"
server_key_iv: "ed4b27d734aef96f0ab8e1f999cd5177"
server_key_version: "encryptionkey@0"
I20260620 23:16:14.340590 32189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:14.356920 32189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:14.360958 32189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:14.361284 32189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:14.379705 32189 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42303
I20260620 23:16:14.379746 32255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42303 every 8 connection(s)
I20260620 23:16:14.381084 32189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:16:14.382122 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 32189
I20260620 23:16:14.382329 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:16:14.382855 11007 external_mini_cluster.cc:1428] Setting key 745ff413ef9241a15aed89501219f4c1
I20260620 23:16:14.385769 32256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:14.395746 32259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:14.384850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.227:48629 (local address 127.0.0.1:42303)
0620 23:16:14.385581 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:14.385593 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:14.385633 (+ 40us) server_negotiation.cc:407] Connection header received
0620 23:16:14.387389 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:14.387409 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:14.387773 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:14.388144 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:14.388687 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:14.390001 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:14.391371 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:14.392036 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:14.392247 (+ 211us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:14.393481 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:14.393489 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:14.393502 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:14.394073 (+ 571us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:14.394142 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:14.394173 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:14.394354 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:14.394633 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 23:16:14.394790 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":344,"threads_started":1}
I20260620 23:16:14.396413 32256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d49c484f2b946ceb77b81635ab759e2: Bootstrap starting.
I20260620 23:16:14.399487 11007 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:16:14.399684 32256 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d49c484f2b946ceb77b81635ab759e2: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:14.400655 32256 log.cc:824] T 00000000000000000000000000000000 P 1d49c484f2b946ceb77b81635ab759e2: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:14.403559 32256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d49c484f2b946ceb77b81635ab759e2: No bootstrap required, opened a new log
I20260620 23:16:14.412454 32256 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d49c484f2b946ceb77b81635ab759e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d49c484f2b946ceb77b81635ab759e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42303 } }
I20260620 23:16:14.412884 32259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:14.403859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.227:40521 (local address 127.0.0.1:42303)
0620 23:16:14.404007 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:14.404015 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:14.407411 (+ 3396us) server_negotiation.cc:407] Connection header received
0620 23:16:14.407617 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:14.407621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:14.407677 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:14.407799 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:14.408270 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:14.409045 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:14.410371 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:14.410833 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:14.411035 (+ 202us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:14.412154 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:14.412159 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:14.412162 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:14.412180 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:14.412214 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:14.412229 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:14.412349 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:14.412541 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 23:16:14.412593 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 23:16:14.412976 32256 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d49c484f2b946ceb77b81635ab759e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:14.413125 32256 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d49c484f2b946ceb77b81635ab759e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d49c484f2b946ceb77b81635ab759e2, State: Initialized, Role: FOLLOWER
I20260620 23:16:14.413738 32256 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d49c484f2b946ceb77b81635ab759e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d49c484f2b946ceb77b81635ab759e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42303 } }
I20260620 23:16:14.413985 32256 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d49c484f2b946ceb77b81635ab759e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:16:14.414166 32256 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d49c484f2b946ceb77b81635ab759e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:16:14.414350 32256 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d49c484f2b946ceb77b81635ab759e2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:14.416863 32256 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d49c484f2b946ceb77b81635ab759e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d49c484f2b946ceb77b81635ab759e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42303 } }
I20260620 23:16:14.417296 32256 leader_election.cc:302] T 00000000000000000000000000000000 P 1d49c484f2b946ceb77b81635ab759e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d49c484f2b946ceb77b81635ab759e2; no voters:
I20260620 23:16:14.417806 32256 leader_election.cc:288] T 00000000000000000000000000000000 P 1d49c484f2b946ceb77b81635ab759e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:16:14.417943 32266 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d49c484f2b946ceb77b81635ab759e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:16:14.418643 32266 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d49c484f2b946ceb77b81635ab759e2 [term 1 LEADER]: Becoming Leader. State: Replica: 1d49c484f2b946ceb77b81635ab759e2, State: Running, Role: LEADER
I20260620 23:16:14.419178 32266 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d49c484f2b946ceb77b81635ab759e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d49c484f2b946ceb77b81635ab759e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42303 } }
I20260620 23:16:14.419812 32256 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d49c484f2b946ceb77b81635ab759e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:16:14.422386 32267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d49c484f2b946ceb77b81635ab759e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d49c484f2b946ceb77b81635ab759e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d49c484f2b946ceb77b81635ab759e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42303 } } }
I20260620 23:16:14.422946 32268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d49c484f2b946ceb77b81635ab759e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d49c484f2b946ceb77b81635ab759e2. Latest consensus state: current_term: 1 leader_uuid: "1d49c484f2b946ceb77b81635ab759e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d49c484f2b946ceb77b81635ab759e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42303 } } }
I20260620 23:16:14.423341 32268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d49c484f2b946ceb77b81635ab759e2 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:14.422955 32267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d49c484f2b946ceb77b81635ab759e2 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:14.424201 32272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:16:14.429339 32272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:16:14.439648 32272 catalog_manager.cc:1347] Generated new cluster ID: 8ac61b2aa79c419c8e78c5352eba0c53
I20260620 23:16:14.439772 32272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:16:14.463470 32272 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:16:14.464895 32272 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:16:14.473836 32272 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d49c484f2b946ceb77b81635ab759e2: Generated new TSK 0
I20260620 23:16:14.474617 32272 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:16:14.480798 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 32189
W20260620 23:16:14.481998 32251 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.227:48629
W20260620 23:16:14.482821 11007 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 32189: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:16:14.483032 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 32189
W20260620 23:16:14.500187 32174 connection.cc:535] client connection to 127.0.0.1:42303 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42303): Connection reset by peer (error 104)
2026-06-20T23:16:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1884 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997042893763-11007-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:16:14 dist-test-slave-jkbq krb5kdc[32287](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:16:14 dist-test-slave-jkbq krb5kdc[32287](info): set up 2 sockets
Jun 20 23:16:14 dist-test-slave-jkbq krb5kdc[32287](info): commencing operation
krb5kdc: starting...
W20260620 23:16:15.583307 11007 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.003s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Jun 20 23:16:15 dist-test-slave-jkbq krb5kdc[32287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:16:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:16:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:16:15.822299 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43867
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:16:16.145953 32300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:16.146258 32300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:16.146453 32300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:16.153822 32300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:16:16.153980 32300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:16.154078 32300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:16.154198 32300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:16:16.154275 32300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:16:16.164279 32300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:16:16.165592 32300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:16.166920 32300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:16.178675 32306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:16.178719 32305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:16.178848 32308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:16.179869 32300 server_base.cc:1034] running on GCE node
I20260620 23:16:16.180680 32300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:16.182022 32300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:16.183274 32300 hybrid_clock.cc:648] HybridClock initialized: now 1781997376183238 us; error 42 us; skew 500 ppm
I20260620 23:16:16.183636 32300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:16:16 dist-test-slave-jkbq krb5kdc[32287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:16.191670 32300 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:16:16.193037 32300 webserver.cc:460] Webserver started at http://127.0.0.1:33545/ using document root <none> and password file <none>
I20260620 23:16:16.193718 32300 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:16.193819 32300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:16.194064 32300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:16.196913 32300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "e55e735de570451984b8c7894e25fcac"
format_stamp: "Formatted at 2026-06-20 23:16:16 on dist-test-slave-jkbq"
server_key: "f086e376b944d160024a6433c8085a8c"
server_key_iv: "2ec076e327bee2e49780e08724f99986"
server_key_version: "encryptionkey@0"
I20260620 23:16:16.197609 32300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "e55e735de570451984b8c7894e25fcac"
format_stamp: "Formatted at 2026-06-20 23:16:16 on dist-test-slave-jkbq"
server_key: "f086e376b944d160024a6433c8085a8c"
server_key_iv: "2ec076e327bee2e49780e08724f99986"
server_key_version: "encryptionkey@0"
I20260620 23:16:16.202481 32300 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 23:16:16.205849 32315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:16.207190 32300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 23:16:16.207347 32300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "e55e735de570451984b8c7894e25fcac"
format_stamp: "Formatted at 2026-06-20 23:16:16 on dist-test-slave-jkbq"
server_key: "f086e376b944d160024a6433c8085a8c"
server_key_iv: "2ec076e327bee2e49780e08724f99986"
server_key_version: "encryptionkey@0"
I20260620 23:16:16.207501 32300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:16.212960 32300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:16.217244 32300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:16.217485 32300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:16.235900 32300 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43867
I20260620 23:16:16.235949 32366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43867 every 8 connection(s)
I20260620 23:16:16.237295 32300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:16:16.241091 32367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:16.244644 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 32300
I20260620 23:16:16.244822 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:16:16.245265 11007 external_mini_cluster.cc:1428] Setting key daacc95c936efb4a28604e19e22270a6
I20260620 23:16:16.254077 32370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:16.247051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.227:39599 (local address 127.0.0.1:43867)
0620 23:16:16.247842 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:16.247856 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:16.247906 (+ 50us) server_negotiation.cc:407] Connection header received
0620 23:16:16.249849 (+ 1943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:16.249880 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:16.250339 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:16.250939 (+ 600us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:16.252065 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:16.252076 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:16.252099 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:16.252117 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:16.252202 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:16.252240 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:16.252473 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:16.252624 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 23:16:16.252833 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":310,"threads_started":1}
I20260620 23:16:16.254983 32367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e55e735de570451984b8c7894e25fcac: Bootstrap starting.
I20260620 23:16:16.257726 11007 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:16:16.258493 32367 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e55e735de570451984b8c7894e25fcac: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:16.259832 32367 log.cc:824] T 00000000000000000000000000000000 P e55e735de570451984b8c7894e25fcac: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:16.263340 32367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e55e735de570451984b8c7894e25fcac: No bootstrap required, opened a new log
I20260620 23:16:16.265821 32370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:16.263387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.227:46555 (local address 127.0.0.1:43867)
0620 23:16:16.263562 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:16.263569 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:16.263879 (+ 310us) server_negotiation.cc:407] Connection header received
0620 23:16:16.263975 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:16.263980 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:16.264044 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:16.264178 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:16.265125 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:16.265141 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:16.265146 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:16.265166 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:16.265219 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:16.265248 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:16.265381 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:16.265432 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 23:16:16.265491 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 23:16:16.272373 32367 raft_consensus.cc:348] T 00000000000000000000000000000000 P e55e735de570451984b8c7894e25fcac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e55e735de570451984b8c7894e25fcac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43867 } }
I20260620 23:16:16.272758 32367 raft_consensus.cc:374] T 00000000000000000000000000000000 P e55e735de570451984b8c7894e25fcac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:16.272877 32367 raft_consensus.cc:729] T 00000000000000000000000000000000 P e55e735de570451984b8c7894e25fcac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e55e735de570451984b8c7894e25fcac, State: Initialized, Role: FOLLOWER
I20260620 23:16:16.273458 32367 consensus_queue.cc:260] T 00000000000000000000000000000000 P e55e735de570451984b8c7894e25fcac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e55e735de570451984b8c7894e25fcac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43867 } }
I20260620 23:16:16.273715 32367 raft_consensus.cc:388] T 00000000000000000000000000000000 P e55e735de570451984b8c7894e25fcac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:16:16.273876 32367 raft_consensus.cc:482] T 00000000000000000000000000000000 P e55e735de570451984b8c7894e25fcac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:16:16.274045 32367 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e55e735de570451984b8c7894e25fcac [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:16.276439 32367 raft_consensus.cc:504] T 00000000000000000000000000000000 P e55e735de570451984b8c7894e25fcac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e55e735de570451984b8c7894e25fcac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43867 } }
I20260620 23:16:16.276957 32367 leader_election.cc:302] T 00000000000000000000000000000000 P e55e735de570451984b8c7894e25fcac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e55e735de570451984b8c7894e25fcac; no voters:
I20260620 23:16:16.277639 32367 leader_election.cc:288] T 00000000000000000000000000000000 P e55e735de570451984b8c7894e25fcac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:16:16.277755 32377 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e55e735de570451984b8c7894e25fcac [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:16:16.278523 32377 raft_consensus.cc:686] T 00000000000000000000000000000000 P e55e735de570451984b8c7894e25fcac [term 1 LEADER]: Becoming Leader. State: Replica: e55e735de570451984b8c7894e25fcac, State: Running, Role: LEADER
I20260620 23:16:16.279101 32377 consensus_queue.cc:237] T 00000000000000000000000000000000 P e55e735de570451984b8c7894e25fcac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e55e735de570451984b8c7894e25fcac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43867 } }
I20260620 23:16:16.279978 32367 sys_catalog.cc:564] T 00000000000000000000000000000000 P e55e735de570451984b8c7894e25fcac [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:16:16.286715 32379 sys_catalog.cc:455] T 00000000000000000000000000000000 P e55e735de570451984b8c7894e25fcac [sys.catalog]: SysCatalogTable state changed. Reason: New leader e55e735de570451984b8c7894e25fcac. Latest consensus state: current_term: 1 leader_uuid: "e55e735de570451984b8c7894e25fcac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e55e735de570451984b8c7894e25fcac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43867 } } }
I20260620 23:16:16.287135 32379 sys_catalog.cc:458] T 00000000000000000000000000000000 P e55e735de570451984b8c7894e25fcac [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:16.289467 32378 sys_catalog.cc:455] T 00000000000000000000000000000000 P e55e735de570451984b8c7894e25fcac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e55e735de570451984b8c7894e25fcac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e55e735de570451984b8c7894e25fcac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43867 } } }
I20260620 23:16:16.289824 32378 sys_catalog.cc:458] T 00000000000000000000000000000000 P e55e735de570451984b8c7894e25fcac [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:16.291304 32382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:16:16.297363 32382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:16:16.306509 32382 catalog_manager.cc:1347] Generated new cluster ID: 9feb3961c917474b8a2a3faf05b65421
I20260620 23:16:16.306679 32382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:16:16.327173 32382 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:16:16.327597 32382 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:16:16.338002 32382 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e55e735de570451984b8c7894e25fcac: Generated new TSK 0
I20260620 23:16:16.339027 32382 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:16:16.345228 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 32300
W20260620 23:16:16.346233 32362 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.227:39599
W20260620 23:16:16.346943 11007 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 32300: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:16:16.347102 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 32300
2026-06-20T23:16:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1864 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997042893763-11007-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:16:16 dist-test-slave-jkbq krb5kdc[32398](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:16:16 dist-test-slave-jkbq krb5kdc[32398](info): set up 2 sockets
Jun 20 23:16:16 dist-test-slave-jkbq krb5kdc[32398](info): commencing operation
krb5kdc: starting...
W20260620 23:16:17.439204 11007 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Jun 20 23:16:17 dist-test-slave-jkbq krb5kdc[32398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:16:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:16:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:16:17.673435 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33259
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:16:17.995997 32411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:17.996269 32411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:17.996428 32411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:18.004076 32411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:16:18.004213 32411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:18.004297 32411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:18.004386 32411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:16:18.004453 32411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:16:18.014248 32411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33259
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:16:18.015604 32411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:18.016896 32411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:18.028776 32416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:18.028851 32417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:18.029021 32419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:18.029965 32411 server_base.cc:1034] running on GCE node
I20260620 23:16:18.030710 32411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:18.032114 32411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:18.033345 32411 hybrid_clock.cc:648] HybridClock initialized: now 1781997378033307 us; error 35 us; skew 500 ppm
I20260620 23:16:18.033684 32411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:16:18 dist-test-slave-jkbq krb5kdc[32398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:18.037163 32411 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:16:18.038614 32411 webserver.cc:460] Webserver started at http://127.0.0.1:32869/ using document root <none> and password file <none>
I20260620 23:16:18.039335 32411 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:18.039471 32411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:18.039780 32411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:18.042699 32411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "a0a639f75bde4c36834ee97889c2525f"
format_stamp: "Formatted at 2026-06-20 23:16:18 on dist-test-slave-jkbq"
server_key: "6e6231f78332064f57b29137c6d9119f"
server_key_iv: "a3f6f6f36f2c21fc920463ede4d7a063"
server_key_version: "encryptionkey@0"
I20260620 23:16:18.043547 32411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "a0a639f75bde4c36834ee97889c2525f"
format_stamp: "Formatted at 2026-06-20 23:16:18 on dist-test-slave-jkbq"
server_key: "6e6231f78332064f57b29137c6d9119f"
server_key_iv: "a3f6f6f36f2c21fc920463ede4d7a063"
server_key_version: "encryptionkey@0"
I20260620 23:16:18.048349 32411 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 23:16:18.051678 32426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:18.052909 32411 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:16:18.053097 32411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "a0a639f75bde4c36834ee97889c2525f"
format_stamp: "Formatted at 2026-06-20 23:16:18 on dist-test-slave-jkbq"
server_key: "6e6231f78332064f57b29137c6d9119f"
server_key_iv: "a3f6f6f36f2c21fc920463ede4d7a063"
server_key_version: "encryptionkey@0"
I20260620 23:16:18.053292 32411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:18.076419 32411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:18.080868 32411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:18.081161 32411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:18.099613 32411 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33259
I20260620 23:16:18.099650 32477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33259 every 8 connection(s)
I20260620 23:16:18.100996 32411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:16:18.104717 32478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:18.105175 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 32411
I20260620 23:16:18.105340 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:16:18.105794 11007 external_mini_cluster.cc:1428] Setting key 44481bdda9182c657d98bb1decf33bb5
I20260620 23:16:18.116693 32478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0a639f75bde4c36834ee97889c2525f: Bootstrap starting.
Jun 20 23:16:18 dist-test-slave-jkbq krb5kdc[32398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 23:16:18.119858 32478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0a639f75bde4c36834ee97889c2525f: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:18.121151 32478 log.cc:824] T 00000000000000000000000000000000 P a0a639f75bde4c36834ee97889c2525f: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:18.124223 32478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0a639f75bde4c36834ee97889c2525f: No bootstrap required, opened a new log
I20260620 23:16:18.128844 32481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:18.107596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.227:40537 (local address 127.0.0.1:33259)
0620 23:16:18.108309 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:18.108327 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:18.108367 (+ 40us) server_negotiation.cc:407] Connection header received
0620 23:16:18.110692 (+ 2325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:18.110714 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:18.111192 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:18.111601 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:18.113005 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:18.114292 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:18.115628 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:18.116289 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:18.116513 (+ 224us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:18.121129 (+ 4616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:18.121137 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:18.121154 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:18.121193 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:18.124524 (+ 3331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:18.125516 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:18.125522 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:18.125527 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:18.125710 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:18.126171 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:18.126174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:18.126176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:18.126678 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:18.127008 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:18.127476 (+ 468us) server_negotiation.cc:299] Negotiation successful
0620 23:16:18.127828 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":325,"threads_started":1}
I20260620 23:16:18.132871 11007 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:16:18.132865 32478 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0a639f75bde4c36834ee97889c2525f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0a639f75bde4c36834ee97889c2525f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33259 } }
I20260620 23:16:18.133287 32478 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0a639f75bde4c36834ee97889c2525f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:18.133383 32478 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0a639f75bde4c36834ee97889c2525f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0a639f75bde4c36834ee97889c2525f, State: Initialized, Role: FOLLOWER
I20260620 23:16:18.133989 32478 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0a639f75bde4c36834ee97889c2525f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0a639f75bde4c36834ee97889c2525f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33259 } }
I20260620 23:16:18.134250 32478 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0a639f75bde4c36834ee97889c2525f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:16:18.134382 32478 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0a639f75bde4c36834ee97889c2525f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:16:18.134531 32478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0a639f75bde4c36834ee97889c2525f [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:18.137863 32478 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0a639f75bde4c36834ee97889c2525f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0a639f75bde4c36834ee97889c2525f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33259 } }
I20260620 23:16:18.138440 32478 leader_election.cc:302] T 00000000000000000000000000000000 P a0a639f75bde4c36834ee97889c2525f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0a639f75bde4c36834ee97889c2525f; no voters:
I20260620 23:16:18.139134 32478 leader_election.cc:288] T 00000000000000000000000000000000 P a0a639f75bde4c36834ee97889c2525f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:16:18.139225 32488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0a639f75bde4c36834ee97889c2525f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:16:18.141814 32478 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0a639f75bde4c36834ee97889c2525f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:16:18.143126 32488 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0a639f75bde4c36834ee97889c2525f [term 1 LEADER]: Becoming Leader. State: Replica: a0a639f75bde4c36834ee97889c2525f, State: Running, Role: LEADER
I20260620 23:16:18.143924 32488 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0a639f75bde4c36834ee97889c2525f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0a639f75bde4c36834ee97889c2525f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33259 } }
I20260620 23:16:18.147490 32489 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0a639f75bde4c36834ee97889c2525f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0a639f75bde4c36834ee97889c2525f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0a639f75bde4c36834ee97889c2525f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33259 } } }
I20260620 23:16:18.147987 32489 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0a639f75bde4c36834ee97889c2525f [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:18.150372 32490 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0a639f75bde4c36834ee97889c2525f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0a639f75bde4c36834ee97889c2525f. Latest consensus state: current_term: 1 leader_uuid: "a0a639f75bde4c36834ee97889c2525f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0a639f75bde4c36834ee97889c2525f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33259 } } }
I20260620 23:16:18.150803 32498 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:16:18.150810 32490 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0a639f75bde4c36834ee97889c2525f [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:18.154961 32481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:18.138010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.227:59205 (local address 127.0.0.1:33259)
0620 23:16:18.138201 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:18.138208 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:18.138527 (+ 319us) server_negotiation.cc:407] Connection header received
0620 23:16:18.138717 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:18.138721 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:18.138782 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:18.138914 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:18.140666 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:18.141547 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:18.142849 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:18.143517 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:18.143657 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:18.147653 (+ 3996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:18.147660 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:18.147664 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:18.147705 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:18.151481 (+ 3776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:18.152432 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:18.152436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:18.152438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:18.152556 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:18.153144 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:18.153152 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:18.153157 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:18.153603 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:18.153884 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:18.154323 (+ 439us) server_negotiation.cc:299] Negotiation successful
0620 23:16:18.154529 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 23:16:18.157159 32498 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:16:18.166035 32498 catalog_manager.cc:1347] Generated new cluster ID: 3646475c2ccc41029ebf780a6ab3e63d
I20260620 23:16:18.166143 32498 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:16:18.183868 32498 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:16:18.185658 32498 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:16:18.198143 32498 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0a639f75bde4c36834ee97889c2525f: Generated new TSK 0
I20260620 23:16:18.198985 32498 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:16:18.207975 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 32411
I20260620 23:16:18.209547 32473 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:16:18.304003 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 32411
W20260620 23:16:18.320935 32396 connection.cc:535] client connection to 127.0.0.1:33259 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33259): Connection reset by peer (error 104)
2026-06-20T23:16:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1956 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997042893763-11007-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:16:18 dist-test-slave-jkbq krb5kdc[32510](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:16:18 dist-test-slave-jkbq krb5kdc[32510](info): set up 2 sockets
Jun 20 23:16:18 dist-test-slave-jkbq krb5kdc[32510](info): commencing operation
krb5kdc: starting...
W20260620 23:16:19.411360 11007 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Jun 20 23:16:19 dist-test-slave-jkbq krb5kdc[32510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:16:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:16:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:16:19.646435 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38669
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:16:19.963951 32523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:19.964299 32523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:19.964466 32523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:19.971673 32523 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:16:19.971823 32523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:19.971939 32523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:19.972047 32523 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:16:19.972126 32523 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:16:19.981788 32523 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38669
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:16:19.983053 32523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:19.984340 32523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:19.995811 32529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:19.995755 32528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:19.995849 32531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:19.996855 32523 server_base.cc:1034] running on GCE node
I20260620 23:16:19.997670 32523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:19.999176 32523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:20.000406 32523 hybrid_clock.cc:648] HybridClock initialized: now 1781997380000359 us; error 38 us; skew 500 ppm
I20260620 23:16:20.000801 32523 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:16:20 dist-test-slave-jkbq krb5kdc[32510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:20.004065 32523 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:16:20.005617 32523 webserver.cc:460] Webserver started at http://127.0.0.1:33885/ using document root <none> and password file <none>
I20260620 23:16:20.006304 32523 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:20.006433 32523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:20.006744 32523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:20.009722 32523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "b94cef06fc3a40b288889eb59468d297"
format_stamp: "Formatted at 2026-06-20 23:16:20 on dist-test-slave-jkbq"
server_key: "11698a90a804a8d36fb84a694042ecbd"
server_key_iv: "f6b3ece20bc69cfed107b17ca825d104"
server_key_version: "encryptionkey@0"
I20260620 23:16:20.010476 32523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "b94cef06fc3a40b288889eb59468d297"
format_stamp: "Formatted at 2026-06-20 23:16:20 on dist-test-slave-jkbq"
server_key: "11698a90a804a8d36fb84a694042ecbd"
server_key_iv: "f6b3ece20bc69cfed107b17ca825d104"
server_key_version: "encryptionkey@0"
I20260620 23:16:20.015511 32523 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260620 23:16:20.019062 32538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:20.020471 32523 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:16:20.020666 32523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "b94cef06fc3a40b288889eb59468d297"
format_stamp: "Formatted at 2026-06-20 23:16:20 on dist-test-slave-jkbq"
server_key: "11698a90a804a8d36fb84a694042ecbd"
server_key_iv: "f6b3ece20bc69cfed107b17ca825d104"
server_key_version: "encryptionkey@0"
I20260620 23:16:20.020861 32523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:20.041571 32523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:20.045576 32523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:20.045866 32523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:20.064663 32523 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38669
I20260620 23:16:20.064711 32589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38669 every 8 connection(s)
I20260620 23:16:20.065980 32523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:16:20.068642 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 32523
I20260620 23:16:20.068825 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:16:20.069382 11007 external_mini_cluster.cc:1428] Setting key 3b43a0ba822e82f9459260436a68c697
I20260620 23:16:20.070132 32590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:20.082628 32593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:20.071181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.227:42089 (local address 127.0.0.1:38669)
0620 23:16:20.071919 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:20.071930 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:20.071970 (+ 40us) server_negotiation.cc:407] Connection header received
0620 23:16:20.073457 (+ 1487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:20.073477 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:20.073862 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:20.074208 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:20.074778 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:20.076151 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:20.078007 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:20.078754 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:20.079005 (+ 251us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:20.080261 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:20.080268 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:20.080285 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:20.080854 (+ 569us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:20.080931 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:20.080960 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:20.081096 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:20.081435 (+ 339us) server_negotiation.cc:299] Negotiation successful
0620 23:16:20.081640 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":355,"threads_started":1}
I20260620 23:16:20.083532 32590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b94cef06fc3a40b288889eb59468d297: Bootstrap starting.
I20260620 23:16:20.086463 11007 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:16:20.086652 32590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b94cef06fc3a40b288889eb59468d297: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:20.087863 32590 log.cc:824] T 00000000000000000000000000000000 P b94cef06fc3a40b288889eb59468d297: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:20.091070 32590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b94cef06fc3a40b288889eb59468d297: No bootstrap required, opened a new log
I20260620 23:16:20.098445 32593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:20.091681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.227:38471 (local address 127.0.0.1:38669)
0620 23:16:20.091878 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:20.091883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:20.092229 (+ 346us) server_negotiation.cc:407] Connection header received
0620 23:16:20.092403 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:20.092406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:20.092462 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:20.092540 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:20.093166 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:20.093948 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:20.095840 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:20.096315 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:20.096411 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:20.097614 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:20.097621 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:20.097624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:20.097640 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:20.097694 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:20.097708 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:20.097794 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:20.098056 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 23:16:20.098113 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 23:16:20.101001 32590 raft_consensus.cc:348] T 00000000000000000000000000000000 P b94cef06fc3a40b288889eb59468d297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b94cef06fc3a40b288889eb59468d297" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38669 } }
I20260620 23:16:20.101369 32590 raft_consensus.cc:374] T 00000000000000000000000000000000 P b94cef06fc3a40b288889eb59468d297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:20.101483 32590 raft_consensus.cc:729] T 00000000000000000000000000000000 P b94cef06fc3a40b288889eb59468d297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b94cef06fc3a40b288889eb59468d297, State: Initialized, Role: FOLLOWER
I20260620 23:16:20.102020 32590 consensus_queue.cc:260] T 00000000000000000000000000000000 P b94cef06fc3a40b288889eb59468d297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b94cef06fc3a40b288889eb59468d297" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38669 } }
I20260620 23:16:20.102308 32590 raft_consensus.cc:388] T 00000000000000000000000000000000 P b94cef06fc3a40b288889eb59468d297 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:16:20.102458 32590 raft_consensus.cc:482] T 00000000000000000000000000000000 P b94cef06fc3a40b288889eb59468d297 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:16:20.102648 32590 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b94cef06fc3a40b288889eb59468d297 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:20.105242 32590 raft_consensus.cc:504] T 00000000000000000000000000000000 P b94cef06fc3a40b288889eb59468d297 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b94cef06fc3a40b288889eb59468d297" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38669 } }
I20260620 23:16:20.105733 32590 leader_election.cc:302] T 00000000000000000000000000000000 P b94cef06fc3a40b288889eb59468d297 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b94cef06fc3a40b288889eb59468d297; no voters:
I20260620 23:16:20.106338 32590 leader_election.cc:288] T 00000000000000000000000000000000 P b94cef06fc3a40b288889eb59468d297 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:16:20.106485 32600 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b94cef06fc3a40b288889eb59468d297 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:16:20.107211 32600 raft_consensus.cc:686] T 00000000000000000000000000000000 P b94cef06fc3a40b288889eb59468d297 [term 1 LEADER]: Becoming Leader. State: Replica: b94cef06fc3a40b288889eb59468d297, State: Running, Role: LEADER
I20260620 23:16:20.107798 32600 consensus_queue.cc:237] T 00000000000000000000000000000000 P b94cef06fc3a40b288889eb59468d297 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b94cef06fc3a40b288889eb59468d297" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38669 } }
I20260620 23:16:20.108407 32590 sys_catalog.cc:564] T 00000000000000000000000000000000 P b94cef06fc3a40b288889eb59468d297 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:16:20.110477 32602 sys_catalog.cc:455] T 00000000000000000000000000000000 P b94cef06fc3a40b288889eb59468d297 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b94cef06fc3a40b288889eb59468d297. Latest consensus state: current_term: 1 leader_uuid: "b94cef06fc3a40b288889eb59468d297" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b94cef06fc3a40b288889eb59468d297" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38669 } } }
I20260620 23:16:20.110916 32602 sys_catalog.cc:458] T 00000000000000000000000000000000 P b94cef06fc3a40b288889eb59468d297 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:20.111075 32601 sys_catalog.cc:455] T 00000000000000000000000000000000 P b94cef06fc3a40b288889eb59468d297 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b94cef06fc3a40b288889eb59468d297" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b94cef06fc3a40b288889eb59468d297" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38669 } } }
I20260620 23:16:20.111438 32601 sys_catalog.cc:458] T 00000000000000000000000000000000 P b94cef06fc3a40b288889eb59468d297 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:20.113567 32606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:16:20.119668 32606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:16:20.129173 32606 catalog_manager.cc:1347] Generated new cluster ID: df6dd9468831472ba0a7648d44b1cbc1
I20260620 23:16:20.129297 32606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:16:20.148202 32606 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:16:20.149685 32606 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:16:20.159323 32606 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b94cef06fc3a40b288889eb59468d297: Generated new TSK 0
I20260620 23:16:20.160149 32606 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:16:20.176903 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 32523
W20260620 23:16:20.177996 32585 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.227:42089
W20260620 23:16:20.178719 11007 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 32523: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:16:20.178894 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 32523
W20260620 23:16:20.195487 32508 connection.cc:535] client connection to 127.0.0.1:38669 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38669): Connection reset by peer (error 104)
2026-06-20T23:16:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1874 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997042893763-11007-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:16:20 dist-test-slave-jkbq krb5kdc[32621](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:16:20 dist-test-slave-jkbq krb5kdc[32621](info): set up 2 sockets
Jun 20 23:16:20 dist-test-slave-jkbq krb5kdc[32621](info): commencing operation
krb5kdc: starting...
W20260620 23:16:21.295380 11007 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.010s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Jun 20 23:16:21 dist-test-slave-jkbq krb5kdc[32621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:16:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:16:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:16:21.541393 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36069
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:16:21.863293 32634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:21.863574 32634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:21.863669 32634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:21.870826 32634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:16:21.870987 32634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:21.871047 32634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:21.871111 32634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:16:21.871155 32634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:16:21.880826 32634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36069
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997042893763-11007-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:16:21.882014 32634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:21.883332 32634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:21.894763 32640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:21.894812 32642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:21.894842 32639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:21.895678 32634 server_base.cc:1034] running on GCE node
I20260620 23:16:21.896406 32634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:21.897720 32634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:21.898970 32634 hybrid_clock.cc:648] HybridClock initialized: now 1781997381898916 us; error 45 us; skew 500 ppm
I20260620 23:16:21.899322 32634 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:16:21 dist-test-slave-jkbq krb5kdc[32621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997381, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:21.906096 32634 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:16:21.907582 32634 webserver.cc:460] Webserver started at http://127.0.0.1:46167/ using document root <none> and password file <none>
I20260620 23:16:21.908257 32634 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:21.908414 32634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:21.908717 32634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:21.911755 32634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "94164aa890b94277b33fcff8e01106c1"
format_stamp: "Formatted at 2026-06-20 23:16:21 on dist-test-slave-jkbq"
server_key: "c81a71bb4569b53a2c9ee7819a7053fe"
server_key_iv: "cdaaf542a5eaaaef85ffe56cd8325fd7"
server_key_version: "encryptionkey@0"
I20260620 23:16:21.912544 32634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "94164aa890b94277b33fcff8e01106c1"
format_stamp: "Formatted at 2026-06-20 23:16:21 on dist-test-slave-jkbq"
server_key: "c81a71bb4569b53a2c9ee7819a7053fe"
server_key_iv: "cdaaf542a5eaaaef85ffe56cd8325fd7"
server_key_version: "encryptionkey@0"
I20260620 23:16:21.917582 32634 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:16:21.920876 32649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:21.922183 32634 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:16:21.922364 32634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "94164aa890b94277b33fcff8e01106c1"
format_stamp: "Formatted at 2026-06-20 23:16:21 on dist-test-slave-jkbq"
server_key: "c81a71bb4569b53a2c9ee7819a7053fe"
server_key_iv: "cdaaf542a5eaaaef85ffe56cd8325fd7"
server_key_version: "encryptionkey@0"
I20260620 23:16:21.922564 32634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:21.928052 32634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:21.931821 32634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:21.932109 32634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:21.950476 32634 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36069
I20260620 23:16:21.950562 32700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36069 every 8 connection(s)
I20260620 23:16:21.951967 32634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:16:21.953354 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 32634
I20260620 23:16:21.953538 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:16:21.953979 11007 external_mini_cluster.cc:1428] Setting key e2305b916f439f1006b4cdabb05a79d4
I20260620 23:16:21.956322 32701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:21.961573 32704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:21.955889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.227:38655 (local address 127.0.0.1:36069)
0620 23:16:21.956707 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:21.956718 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:21.956757 (+ 39us) server_negotiation.cc:407] Connection header received
0620 23:16:21.958324 (+ 1567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:21.958344 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:21.958671 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:21.959064 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:21.960078 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:21.960084 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:21.960096 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:21.960113 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:21.960182 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:21.960201 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:21.960371 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:21.960471 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 23:16:21.960672 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":365,"threads_started":1}
I20260620 23:16:21.965083 11007 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:16:21.968911 32701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94164aa890b94277b33fcff8e01106c1: Bootstrap starting.
I20260620 23:16:21.972501 32701 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94164aa890b94277b33fcff8e01106c1: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:21.973167 32704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:21.970394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.227:42603 (local address 127.0.0.1:36069)
0620 23:16:21.970602 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:21.970611 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:21.971156 (+ 545us) server_negotiation.cc:407] Connection header received
0620 23:16:21.971293 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:21.971297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:21.971351 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:21.971459 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:21.972415 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:21.972420 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:21.972425 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:21.972443 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:21.972506 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:21.972527 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:21.972641 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:21.972796 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 23:16:21.972861 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:16:21.973719 32701 log.cc:824] T 00000000000000000000000000000000 P 94164aa890b94277b33fcff8e01106c1: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:21.976306 32701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94164aa890b94277b33fcff8e01106c1: No bootstrap required, opened a new log
I20260620 23:16:21.983203 32701 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94164aa890b94277b33fcff8e01106c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94164aa890b94277b33fcff8e01106c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36069 } }
I20260620 23:16:21.983559 32701 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94164aa890b94277b33fcff8e01106c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:21.983668 32701 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94164aa890b94277b33fcff8e01106c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94164aa890b94277b33fcff8e01106c1, State: Initialized, Role: FOLLOWER
I20260620 23:16:21.984251 32701 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94164aa890b94277b33fcff8e01106c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94164aa890b94277b33fcff8e01106c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36069 } }
I20260620 23:16:21.984481 32701 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94164aa890b94277b33fcff8e01106c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:16:21.984613 32701 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94164aa890b94277b33fcff8e01106c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:16:21.984789 32701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94164aa890b94277b33fcff8e01106c1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:21.986922 32701 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94164aa890b94277b33fcff8e01106c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94164aa890b94277b33fcff8e01106c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36069 } }
I20260620 23:16:21.987421 32701 leader_election.cc:302] T 00000000000000000000000000000000 P 94164aa890b94277b33fcff8e01106c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94164aa890b94277b33fcff8e01106c1; no voters:
I20260620 23:16:21.988006 32701 leader_election.cc:288] T 00000000000000000000000000000000 P 94164aa890b94277b33fcff8e01106c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:16:21.988109 32711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94164aa890b94277b33fcff8e01106c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:16:21.988878 32711 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94164aa890b94277b33fcff8e01106c1 [term 1 LEADER]: Becoming Leader. State: Replica: 94164aa890b94277b33fcff8e01106c1, State: Running, Role: LEADER
I20260620 23:16:21.989449 32711 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94164aa890b94277b33fcff8e01106c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94164aa890b94277b33fcff8e01106c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36069 } }
I20260620 23:16:21.990254 32701 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94164aa890b94277b33fcff8e01106c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:16:21.992581 32712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94164aa890b94277b33fcff8e01106c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94164aa890b94277b33fcff8e01106c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94164aa890b94277b33fcff8e01106c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36069 } } }
I20260620 23:16:21.993072 32712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94164aa890b94277b33fcff8e01106c1 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:21.994371 32713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94164aa890b94277b33fcff8e01106c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94164aa890b94277b33fcff8e01106c1. Latest consensus state: current_term: 1 leader_uuid: "94164aa890b94277b33fcff8e01106c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94164aa890b94277b33fcff8e01106c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36069 } } }
I20260620 23:16:21.999115 32713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94164aa890b94277b33fcff8e01106c1 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:21.999858 32720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:16:22.005074 32720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:16:22.013870 32720 catalog_manager.cc:1347] Generated new cluster ID: bf7bd62136424082913f87f75fcbf21f
I20260620 23:16:22.014001 32720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:16:22.031324 32720 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:16:22.031786 32720 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:16:22.041509 32720 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94164aa890b94277b33fcff8e01106c1: Generated new TSK 0
I20260620 23:16:22.042486 32720 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:16:22.046949 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 32634
W20260620 23:16:22.048157 32696 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.227:38655
W20260620 23:16:22.048918 11007 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 32634: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:16:22.049070 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 32634
2026-06-20T23:16:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1867 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997042893763-11007-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:16:22 dist-test-slave-jkbq krb5kdc[32732](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:16:22 dist-test-slave-jkbq krb5kdc[32732](info): set up 2 sockets
Jun 20 23:16:22 dist-test-slave-jkbq krb5kdc[32732](info): commencing operation
krb5kdc: starting...
W20260620 23:16:23.147360 11007 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Jun 20 23:16:23 dist-test-slave-jkbq krb5kdc[32732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:16:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:16:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.191.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
I20260620 23:16:23.394062 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:40195
--webserver_interface=127.10.191.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:38635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:40195
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:16:23.719292 32745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:23.719619 32745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:23.719800 32745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:23.726902 32745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:16:23.727062 32745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:23.727169 32745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:23.727288 32745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:16:23.727367 32745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:16:23.737064 32745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:38635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:40195
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:40195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:16:23.738375 32745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:23.739715 32745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:23.751885 32750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:23.751832 32751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:23.751946 32753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:23.752796 32745 server_base.cc:1034] running on GCE node
I20260620 23:16:23.753569 32745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:23.755053 32745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:23.756294 32745 hybrid_clock.cc:648] HybridClock initialized: now 1781997383756251 us; error 48 us; skew 500 ppm
I20260620 23:16:23.756644 32745 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:16:23 dist-test-slave-jkbq krb5kdc[32732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997383, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:23.759847 32745 init.cc:377] Logged in from keytab as kudu/127.10.191.254@KRBTEST.COM (short username kudu)
I20260620 23:16:23.761420 32745 webserver.cc:460] Webserver started at http://127.10.191.254:39509/ using document root <none> and password file <none>
I20260620 23:16:23.762130 32745 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:23.762282 32745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:23.762584 32745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:23.765602 32745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "85b7fd41486f492d80a9cfb5f1de3e85"
format_stamp: "Formatted at 2026-06-20 23:16:23 on dist-test-slave-jkbq"
server_key: "d404d45fcf61cf4277e87a58cc599e20"
server_key_iv: "a2d994ddf7a8dcc042b4ebb01fb3bf9e"
server_key_version: "encryptionkey@0"
I20260620 23:16:23.766361 32745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "85b7fd41486f492d80a9cfb5f1de3e85"
format_stamp: "Formatted at 2026-06-20 23:16:23 on dist-test-slave-jkbq"
server_key: "d404d45fcf61cf4277e87a58cc599e20"
server_key_iv: "a2d994ddf7a8dcc042b4ebb01fb3bf9e"
server_key_version: "encryptionkey@0"
I20260620 23:16:23.771422 32745 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 23:16:23.774755 32760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:23.776043 32745 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:16:23.776244 32745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "85b7fd41486f492d80a9cfb5f1de3e85"
format_stamp: "Formatted at 2026-06-20 23:16:23 on dist-test-slave-jkbq"
server_key: "d404d45fcf61cf4277e87a58cc599e20"
server_key_iv: "a2d994ddf7a8dcc042b4ebb01fb3bf9e"
server_key_version: "encryptionkey@0"
I20260620 23:16:23.776433 32745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:23.789831 32745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:23.793605 32745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:23.793880 32745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:23.812402 32745 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:40195
I20260620 23:16:23.812425 343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:40195 every 8 connection(s)
I20260620 23:16:23.813838 32745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:16:23.817051 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 32745
I20260620 23:16:23.817234 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:16:23.817780 11007 external_mini_cluster.cc:1428] Setting key fe2efe75e54be5685dc25072e673b40a
I20260620 23:16:23.817865 344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:23.832389 344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85b7fd41486f492d80a9cfb5f1de3e85: Bootstrap starting.
Jun 20 23:16:23 dist-test-slave-jkbq krb5kdc[32732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:16:23.835942 344 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85b7fd41486f492d80a9cfb5f1de3e85: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:23.837049 344 log.cc:824] T 00000000000000000000000000000000 P 85b7fd41486f492d80a9cfb5f1de3e85: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:23.839933 344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85b7fd41486f492d80a9cfb5f1de3e85: No bootstrap required, opened a new log
I20260620 23:16:23.841279 347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:23.819535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45731 (local address 127.10.191.254:40195)
0620 23:16:23.820309 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:23.820319 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:23.820360 (+ 41us) server_negotiation.cc:407] Connection header received
0620 23:16:23.822359 (+ 1999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:23.822380 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:23.822770 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:23.823164 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:23.824872 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:23.826141 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:23.827981 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:23.828737 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:23.828897 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:23.833865 (+ 4968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:23.833876 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:23.833890 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:23.833936 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:23.837076 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:23.837917 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:23.837923 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:23.837929 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:23.838111 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:23.838598 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:23.838602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:23.838604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:23.839168 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:23.839494 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:23.839967 (+ 473us) server_negotiation.cc:299] Negotiation successful
0620 23:16:23.840318 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":361,"threads_started":1}
I20260620 23:16:23.845361 11007 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:16:23.848505 344 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85b7fd41486f492d80a9cfb5f1de3e85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b7fd41486f492d80a9cfb5f1de3e85" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 40195 } }
I20260620 23:16:23.848886 344 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85b7fd41486f492d80a9cfb5f1de3e85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:23.848991 344 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85b7fd41486f492d80a9cfb5f1de3e85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85b7fd41486f492d80a9cfb5f1de3e85, State: Initialized, Role: FOLLOWER
I20260620 23:16:23.849646 344 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85b7fd41486f492d80a9cfb5f1de3e85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b7fd41486f492d80a9cfb5f1de3e85" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 40195 } }
I20260620 23:16:23.849997 344 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85b7fd41486f492d80a9cfb5f1de3e85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:16:23.850153 344 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85b7fd41486f492d80a9cfb5f1de3e85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:16:23.850329 344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85b7fd41486f492d80a9cfb5f1de3e85 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:23.853021 344 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85b7fd41486f492d80a9cfb5f1de3e85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b7fd41486f492d80a9cfb5f1de3e85" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 40195 } }
I20260620 23:16:23.853662 344 leader_election.cc:302] T 00000000000000000000000000000000 P 85b7fd41486f492d80a9cfb5f1de3e85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85b7fd41486f492d80a9cfb5f1de3e85; no voters:
I20260620 23:16:23.854332 344 leader_election.cc:288] T 00000000000000000000000000000000 P 85b7fd41486f492d80a9cfb5f1de3e85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:16:23.854432 354 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85b7fd41486f492d80a9cfb5f1de3e85 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:16:23.855870 354 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85b7fd41486f492d80a9cfb5f1de3e85 [term 1 LEADER]: Becoming Leader. State: Replica: 85b7fd41486f492d80a9cfb5f1de3e85, State: Running, Role: LEADER
I20260620 23:16:23.856936 344 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85b7fd41486f492d80a9cfb5f1de3e85 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:16:23.856810 354 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85b7fd41486f492d80a9cfb5f1de3e85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b7fd41486f492d80a9cfb5f1de3e85" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 40195 } }
I20260620 23:16:23.863945 356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85b7fd41486f492d80a9cfb5f1de3e85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85b7fd41486f492d80a9cfb5f1de3e85. Latest consensus state: current_term: 1 leader_uuid: "85b7fd41486f492d80a9cfb5f1de3e85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b7fd41486f492d80a9cfb5f1de3e85" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 40195 } } }
I20260620 23:16:23.864499 356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85b7fd41486f492d80a9cfb5f1de3e85 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:23.864583 355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85b7fd41486f492d80a9cfb5f1de3e85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85b7fd41486f492d80a9cfb5f1de3e85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b7fd41486f492d80a9cfb5f1de3e85" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 40195 } } }
I20260620 23:16:23.865159 355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85b7fd41486f492d80a9cfb5f1de3e85 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:23.866633 347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:23.849645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33815 (local address 127.10.191.254:40195)
0620 23:16:23.849857 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:23.849865 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:23.850332 (+ 467us) server_negotiation.cc:407] Connection header received
0620 23:16:23.850522 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:23.850528 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:23.850614 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:23.850723 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:23.852180 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:23.853285 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:23.854558 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:23.855346 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:23.855478 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:23.858088 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:23.858093 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:23.858097 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:23.858127 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:23.862189 (+ 4062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:23.863410 (+ 1221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:23.863418 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:23.863423 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:23.863623 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:23.864297 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:23.864304 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:23.864310 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:23.864832 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:23.866039 (+ 1207us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:23.866171 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 23:16:23.866311 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":68}
W20260620 23:16:23.869675 368 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 85b7fd41486f492d80a9cfb5f1de3e85: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:16:23.869879 368 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:16:23.875413 369 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:16:23.877192 369 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:16:23.886337 369 catalog_manager.cc:1347] Generated new cluster ID: 6404e03d061b4f8abaf5e6240cfb9501
I20260620 23:16:23.886457 369 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:16:23.906528 369 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:16:23.908506 369 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:16:23.921639 369 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85b7fd41486f492d80a9cfb5f1de3e85: Generated new TSK 0
I20260620 23:16:23.922696 369 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:16:23.931703 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 32745
I20260620 23:16:23.933326 339 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:16:24.026924 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 32745
W20260620 23:16:24.044991 32730 connection.cc:535] client connection to 127.10.191.254:40195 recv error: Network error: failed to read from TLS socket (remote: 127.10.191.254:40195): Connection reset by peer (error 104)
2026-06-20T23:16:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1982 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997042893763-11007-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:16:24 dist-test-slave-jkbq krb5kdc[376](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:16:24 dist-test-slave-jkbq krb5kdc[376](info): set up 2 sockets
Jun 20 23:16:24 dist-test-slave-jkbq krb5kdc[376](info): commencing operation
krb5kdc: starting...
W20260620 23:16:25.135351 11007 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Jun 20 23:16:25 dist-test-slave-jkbq krb5kdc[376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:16:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:16:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.191.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
I20260620 23:16:25.393625 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:45217
--webserver_interface=127.10.191.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:35179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:45217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:16:25.720333 389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:25.720605 389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:25.720790 389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:25.727912 389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:16:25.728041 389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:25.728152 389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:25.728255 389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:16:25.728332 389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:16:25.737970 389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:35179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:45217
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:45217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:16:25.739272 389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:25.740605 389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:25.752266 395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:25.752272 394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:25.752272 397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:25.752959 389 server_base.cc:1034] running on GCE node
I20260620 23:16:25.753762 389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:25.755173 389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:25.756421 389 hybrid_clock.cc:648] HybridClock initialized: now 1781997385756368 us; error 57 us; skew 500 ppm
I20260620 23:16:25.756762 389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:16:25 dist-test-slave-jkbq krb5kdc[376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997385, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:25.759876 389 init.cc:377] Logged in from keytab as kudu/127.10.191.254@KRBTEST.COM (short username kudu)
I20260620 23:16:25.761373 389 webserver.cc:460] Webserver started at http://127.10.191.254:35293/ using document root <none> and password file <none>
I20260620 23:16:25.762054 389 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:25.762210 389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:25.762521 389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:25.765605 389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "c332ccfd6fda483fa41ad3f5227105f7"
format_stamp: "Formatted at 2026-06-20 23:16:25 on dist-test-slave-jkbq"
server_key: "8cc981b42b429bf106cfe35a00825c48"
server_key_iv: "bba1970856faba4087f7431917e7b672"
server_key_version: "encryptionkey@0"
I20260620 23:16:25.766407 389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "c332ccfd6fda483fa41ad3f5227105f7"
format_stamp: "Formatted at 2026-06-20 23:16:25 on dist-test-slave-jkbq"
server_key: "8cc981b42b429bf106cfe35a00825c48"
server_key_iv: "bba1970856faba4087f7431917e7b672"
server_key_version: "encryptionkey@0"
I20260620 23:16:25.771456 389 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 23:16:25.774842 404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:25.776165 389 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:16:25.776350 389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "c332ccfd6fda483fa41ad3f5227105f7"
format_stamp: "Formatted at 2026-06-20 23:16:25 on dist-test-slave-jkbq"
server_key: "8cc981b42b429bf106cfe35a00825c48"
server_key_iv: "bba1970856faba4087f7431917e7b672"
server_key_version: "encryptionkey@0"
I20260620 23:16:25.776533 389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:25.796039 389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:25.800045 389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:25.800328 389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:25.818939 389 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:45217
I20260620 23:16:25.818941 455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:45217 every 8 connection(s)
I20260620 23:16:25.820353 389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:16:25.824033 456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:25.827716 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 389
I20260620 23:16:25.827899 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:16:25.828333 11007 external_mini_cluster.cc:1428] Setting key a6e3ab9e0168b1db2ce5c9702aa87662
I20260620 23:16:25.837180 456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c332ccfd6fda483fa41ad3f5227105f7: Bootstrap starting.
I20260620 23:16:25.840970 456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c332ccfd6fda483fa41ad3f5227105f7: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:25.842301 456 log.cc:824] T 00000000000000000000000000000000 P c332ccfd6fda483fa41ad3f5227105f7: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:25.843000 459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:25.830157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52553 (local address 127.10.191.254:45217)
0620 23:16:25.830949 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:25.830963 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:25.831016 (+ 53us) server_negotiation.cc:407] Connection header received
0620 23:16:25.832748 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:25.832776 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:25.833307 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:25.833739 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:25.834268 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:25.836305 (+ 2037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:25.837686 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:25.838668 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:25.838915 (+ 247us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:25.840069 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:25.840082 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:25.840099 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:25.840768 (+ 669us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:25.840886 (+ 118us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:25.840921 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:25.841096 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:25.841418 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 23:16:25.841593 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":324,"threads_started":1}
I20260620 23:16:25.845218 456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c332ccfd6fda483fa41ad3f5227105f7: No bootstrap required, opened a new log
I20260620 23:16:25.846738 11007 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:16:25.855377 456 raft_consensus.cc:348] T 00000000000000000000000000000000 P c332ccfd6fda483fa41ad3f5227105f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c332ccfd6fda483fa41ad3f5227105f7" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 45217 } }
I20260620 23:16:25.855798 456 raft_consensus.cc:374] T 00000000000000000000000000000000 P c332ccfd6fda483fa41ad3f5227105f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:25.855904 456 raft_consensus.cc:729] T 00000000000000000000000000000000 P c332ccfd6fda483fa41ad3f5227105f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c332ccfd6fda483fa41ad3f5227105f7, State: Initialized, Role: FOLLOWER
I20260620 23:16:25.856592 456 consensus_queue.cc:260] T 00000000000000000000000000000000 P c332ccfd6fda483fa41ad3f5227105f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c332ccfd6fda483fa41ad3f5227105f7" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 45217 } }
I20260620 23:16:25.856859 456 raft_consensus.cc:388] T 00000000000000000000000000000000 P c332ccfd6fda483fa41ad3f5227105f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:16:25.857015 456 raft_consensus.cc:482] T 00000000000000000000000000000000 P c332ccfd6fda483fa41ad3f5227105f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:16:25.857198 456 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c332ccfd6fda483fa41ad3f5227105f7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:25.858417 459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:25.851796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59883 (local address 127.10.191.254:45217)
0620 23:16:25.851975 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:25.851981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:25.852315 (+ 334us) server_negotiation.cc:407] Connection header received
0620 23:16:25.852464 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:25.852474 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:25.852552 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:25.852653 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:25.853105 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:25.854271 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:25.855689 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:25.856355 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:25.856479 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:25.857631 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:25.857637 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:25.857641 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:25.857661 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:25.857727 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:25.857754 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:25.857892 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:25.858031 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 23:16:25.858088 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 23:16:25.860319 456 raft_consensus.cc:504] T 00000000000000000000000000000000 P c332ccfd6fda483fa41ad3f5227105f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c332ccfd6fda483fa41ad3f5227105f7" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 45217 } }
I20260620 23:16:25.860823 456 leader_election.cc:302] T 00000000000000000000000000000000 P c332ccfd6fda483fa41ad3f5227105f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c332ccfd6fda483fa41ad3f5227105f7; no voters:
I20260620 23:16:25.861400 456 leader_election.cc:288] T 00000000000000000000000000000000 P c332ccfd6fda483fa41ad3f5227105f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:16:25.861534 466 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c332ccfd6fda483fa41ad3f5227105f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:16:25.862295 466 raft_consensus.cc:686] T 00000000000000000000000000000000 P c332ccfd6fda483fa41ad3f5227105f7 [term 1 LEADER]: Becoming Leader. State: Replica: c332ccfd6fda483fa41ad3f5227105f7, State: Running, Role: LEADER
I20260620 23:16:25.863025 466 consensus_queue.cc:237] T 00000000000000000000000000000000 P c332ccfd6fda483fa41ad3f5227105f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c332ccfd6fda483fa41ad3f5227105f7" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 45217 } }
I20260620 23:16:25.863682 456 sys_catalog.cc:564] T 00000000000000000000000000000000 P c332ccfd6fda483fa41ad3f5227105f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:16:25.865919 468 sys_catalog.cc:455] T 00000000000000000000000000000000 P c332ccfd6fda483fa41ad3f5227105f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c332ccfd6fda483fa41ad3f5227105f7. Latest consensus state: current_term: 1 leader_uuid: "c332ccfd6fda483fa41ad3f5227105f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c332ccfd6fda483fa41ad3f5227105f7" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 45217 } } }
I20260620 23:16:25.866089 467 sys_catalog.cc:455] T 00000000000000000000000000000000 P c332ccfd6fda483fa41ad3f5227105f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c332ccfd6fda483fa41ad3f5227105f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c332ccfd6fda483fa41ad3f5227105f7" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 45217 } } }
I20260620 23:16:25.866340 468 sys_catalog.cc:458] T 00000000000000000000000000000000 P c332ccfd6fda483fa41ad3f5227105f7 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:25.866344 467 sys_catalog.cc:458] T 00000000000000000000000000000000 P c332ccfd6fda483fa41ad3f5227105f7 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:25.868199 475 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:16:25.873366 475 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:16:25.882447 475 catalog_manager.cc:1347] Generated new cluster ID: 33de2789650a4388b038e9d94af3c551
I20260620 23:16:25.882568 475 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:16:25.899638 475 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:16:25.901358 475 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:16:25.918509 475 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c332ccfd6fda483fa41ad3f5227105f7: Generated new TSK 0
I20260620 23:16:25.919417 475 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:16:25.936851 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 389
W20260620 23:16:25.938334 451 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52553
W20260620 23:16:25.939145 11007 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 389: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:16:25.939317 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 389
W20260620 23:16:25.956282 374 connection.cc:535] client connection to 127.10.191.254:45217 recv error: Network error: failed to read from TLS socket (remote: 127.10.191.254:45217): Connection reset by peer (error 104)
2026-06-20T23:16:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1910 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997042893763-11007-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:16:26 dist-test-slave-jkbq krb5kdc[487](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:16:26 dist-test-slave-jkbq krb5kdc[487](info): set up 2 sockets
Jun 20 23:16:26 dist-test-slave-jkbq krb5kdc[487](info): commencing operation
krb5kdc: starting...
W20260620 23:16:27.047390 11007 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Jun 20 23:16:27 dist-test-slave-jkbq krb5kdc[487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:16:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:16:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.191.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
I20260620 23:16:27.293848 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:43215
--webserver_interface=127.10.191.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:34885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:43215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:16:27.618510 500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:27.618844 500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:27.619038 500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:27.626166 500 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:16:27.626374 500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:27.626479 500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:27.626593 500 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:16:27.626677 500 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:16:27.636485 500 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:34885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:43215
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:43215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:16:27.637811 500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:27.639155 500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:27.651294 508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:27.651356 505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:27.651332 506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:27.652066 500 server_base.cc:1034] running on GCE node
I20260620 23:16:27.652845 500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:27.654242 500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:27.655440 500 hybrid_clock.cc:648] HybridClock initialized: now 1781997387655394 us; error 42 us; skew 500 ppm
I20260620 23:16:27.655805 500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:16:27 dist-test-slave-jkbq krb5kdc[487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997387, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:27.666455 500 init.cc:377] Logged in from keytab as kudu/127.10.191.254@KRBTEST.COM (short username kudu)
I20260620 23:16:27.667975 500 webserver.cc:460] Webserver started at http://127.10.191.254:41965/ using document root <none> and password file <none>
I20260620 23:16:27.668635 500 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:27.668738 500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:27.669001 500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:27.672056 500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "95c79da0a9744f249e5aa833e76d6ec1"
format_stamp: "Formatted at 2026-06-20 23:16:27 on dist-test-slave-jkbq"
server_key: "5d8c783c5678d576955ba7fee6e1ca2b"
server_key_iv: "9bd906edbfcfa5390041b80deba9a532"
server_key_version: "encryptionkey@0"
I20260620 23:16:27.672767 500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "95c79da0a9744f249e5aa833e76d6ec1"
format_stamp: "Formatted at 2026-06-20 23:16:27 on dist-test-slave-jkbq"
server_key: "5d8c783c5678d576955ba7fee6e1ca2b"
server_key_iv: "9bd906edbfcfa5390041b80deba9a532"
server_key_version: "encryptionkey@0"
I20260620 23:16:27.678224 500 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:16:27.681749 515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:27.683189 500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:16:27.683329 500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "95c79da0a9744f249e5aa833e76d6ec1"
format_stamp: "Formatted at 2026-06-20 23:16:27 on dist-test-slave-jkbq"
server_key: "5d8c783c5678d576955ba7fee6e1ca2b"
server_key_iv: "9bd906edbfcfa5390041b80deba9a532"
server_key_version: "encryptionkey@0"
I20260620 23:16:27.683496 500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:27.688930 500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:27.693197 500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:27.693439 500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:27.712212 500 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:43215
I20260620 23:16:27.712292 566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:43215 every 8 connection(s)
I20260620 23:16:27.713608 500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:16:27.716904 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 500
I20260620 23:16:27.717087 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:16:27.717557 11007 external_mini_cluster.cc:1428] Setting key 77a652167c52ff5cbf718dd4cccbe001
I20260620 23:16:27.717590 567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:27.724960 570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:27.719169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34423 (local address 127.10.191.254:43215)
0620 23:16:27.719974 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:27.719985 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:27.720025 (+ 40us) server_negotiation.cc:407] Connection header received
0620 23:16:27.721530 (+ 1505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:27.721549 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:27.721891 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:27.722317 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:27.723366 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:27.723372 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:27.723384 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:27.723398 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:27.723471 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:27.723493 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:27.723733 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:27.723878 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 23:16:27.724057 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":382,"threads_started":1}
I20260620 23:16:27.728927 11007 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:16:27.731210 567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95c79da0a9744f249e5aa833e76d6ec1: Bootstrap starting.
I20260620 23:16:27.734513 567 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95c79da0a9744f249e5aa833e76d6ec1: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:27.735924 570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:27.733331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48937 (local address 127.10.191.254:43215)
0620 23:16:27.733540 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:27.733549 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:27.734019 (+ 470us) server_negotiation.cc:407] Connection header received
0620 23:16:27.734181 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:27.734185 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:27.734238 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:27.734317 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:27.735279 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:27.735283 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:27.735286 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:27.735299 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:27.735334 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:27.735348 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:27.735434 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:27.735580 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 23:16:27.735621 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:16:27.735908 567 log.cc:824] T 00000000000000000000000000000000 P 95c79da0a9744f249e5aa833e76d6ec1: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:27.738790 567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95c79da0a9744f249e5aa833e76d6ec1: No bootstrap required, opened a new log
I20260620 23:16:27.746256 567 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95c79da0a9744f249e5aa833e76d6ec1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95c79da0a9744f249e5aa833e76d6ec1" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43215 } }
I20260620 23:16:27.746613 567 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95c79da0a9744f249e5aa833e76d6ec1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:27.746692 567 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95c79da0a9744f249e5aa833e76d6ec1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95c79da0a9744f249e5aa833e76d6ec1, State: Initialized, Role: FOLLOWER
I20260620 23:16:27.747282 567 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95c79da0a9744f249e5aa833e76d6ec1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95c79da0a9744f249e5aa833e76d6ec1" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43215 } }
I20260620 23:16:27.747488 567 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95c79da0a9744f249e5aa833e76d6ec1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:16:27.747577 567 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95c79da0a9744f249e5aa833e76d6ec1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:16:27.747702 567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95c79da0a9744f249e5aa833e76d6ec1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:27.749759 567 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95c79da0a9744f249e5aa833e76d6ec1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95c79da0a9744f249e5aa833e76d6ec1" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43215 } }
I20260620 23:16:27.750190 567 leader_election.cc:302] T 00000000000000000000000000000000 P 95c79da0a9744f249e5aa833e76d6ec1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95c79da0a9744f249e5aa833e76d6ec1; no voters:
I20260620 23:16:27.750985 567 leader_election.cc:288] T 00000000000000000000000000000000 P 95c79da0a9744f249e5aa833e76d6ec1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:16:27.751093 577 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95c79da0a9744f249e5aa833e76d6ec1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:16:27.751801 577 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95c79da0a9744f249e5aa833e76d6ec1 [term 1 LEADER]: Becoming Leader. State: Replica: 95c79da0a9744f249e5aa833e76d6ec1, State: Running, Role: LEADER
I20260620 23:16:27.752355 577 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95c79da0a9744f249e5aa833e76d6ec1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95c79da0a9744f249e5aa833e76d6ec1" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43215 } }
I20260620 23:16:27.753072 567 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95c79da0a9744f249e5aa833e76d6ec1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:16:27.755407 578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95c79da0a9744f249e5aa833e76d6ec1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95c79da0a9744f249e5aa833e76d6ec1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95c79da0a9744f249e5aa833e76d6ec1" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43215 } } }
I20260620 23:16:27.755901 578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95c79da0a9744f249e5aa833e76d6ec1 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:27.756070 579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95c79da0a9744f249e5aa833e76d6ec1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95c79da0a9744f249e5aa833e76d6ec1. Latest consensus state: current_term: 1 leader_uuid: "95c79da0a9744f249e5aa833e76d6ec1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95c79da0a9744f249e5aa833e76d6ec1" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43215 } } }
I20260620 23:16:27.756434 579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95c79da0a9744f249e5aa833e76d6ec1 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:27.758332 586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:16:27.763926 586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:16:27.772946 586 catalog_manager.cc:1347] Generated new cluster ID: 21f6c2e3a4dd4a2dbf6ec4de978bc944
I20260620 23:16:27.773063 586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:16:27.781538 586 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:16:27.781920 586 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:16:27.787637 586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95c79da0a9744f249e5aa833e76d6ec1: Generated new TSK 0
I20260620 23:16:27.788421 586 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:16:27.802433 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 500
W20260620 23:16:27.803611 562 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34423
W20260620 23:16:27.804281 11007 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 500: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:16:27.804435 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 500
2026-06-20T23:16:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1862 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997042893763-11007-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:16:27 dist-test-slave-jkbq krb5kdc[598](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:16:27 dist-test-slave-jkbq krb5kdc[598](info): set up 2 sockets
Jun 20 23:16:27 dist-test-slave-jkbq krb5kdc[598](info): commencing operation
krb5kdc: starting...
W20260620 23:16:28.927361 11007 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.004s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Jun 20 23:16:29 dist-test-slave-jkbq krb5kdc[598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:16:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:16:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.191.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
I20260620 23:16:29.190435 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:36835
--webserver_interface=127.10.191.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:40683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:36835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:16:29.516805 611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:29.517066 611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:29.517274 611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:29.524629 611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:16:29.524758 611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:29.524871 611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:29.524977 611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:16:29.525059 611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:16:29.534685 611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:40683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:36835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:36835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:16:29.535948 611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:29.537247 611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:29.548995 616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:29.548908 619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:29.548947 617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:29.549912 611 server_base.cc:1034] running on GCE node
I20260620 23:16:29.550663 611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:29.552130 611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:29.553380 611 hybrid_clock.cc:648] HybridClock initialized: now 1781997389553347 us; error 48 us; skew 500 ppm
I20260620 23:16:29.553745 611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:16:29 dist-test-slave-jkbq krb5kdc[598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997389, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:29.556825 611 init.cc:377] Logged in from keytab as kudu/127.10.191.254@KRBTEST.COM (short username kudu)
I20260620 23:16:29.558364 611 webserver.cc:460] Webserver started at http://127.10.191.254:40977/ using document root <none> and password file <none>
I20260620 23:16:29.559108 611 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:29.559259 611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:29.559669 611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:29.563807 611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "0d0a0cb1bcc940739718c757c3dfce9c"
format_stamp: "Formatted at 2026-06-20 23:16:29 on dist-test-slave-jkbq"
server_key: "186eb5cff255207a4786b1f5a9925744"
server_key_iv: "24c5a1bbc134663d9265c060c8577f82"
server_key_version: "encryptionkey@0"
I20260620 23:16:29.564582 611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "0d0a0cb1bcc940739718c757c3dfce9c"
format_stamp: "Formatted at 2026-06-20 23:16:29 on dist-test-slave-jkbq"
server_key: "186eb5cff255207a4786b1f5a9925744"
server_key_iv: "24c5a1bbc134663d9265c060c8577f82"
server_key_version: "encryptionkey@0"
I20260620 23:16:29.569736 611 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260620 23:16:29.573421 626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:29.574747 611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 23:16:29.575008 611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "0d0a0cb1bcc940739718c757c3dfce9c"
format_stamp: "Formatted at 2026-06-20 23:16:29 on dist-test-slave-jkbq"
server_key: "186eb5cff255207a4786b1f5a9925744"
server_key_iv: "24c5a1bbc134663d9265c060c8577f82"
server_key_version: "encryptionkey@0"
I20260620 23:16:29.575228 611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:29.591715 611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:29.595793 611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:29.596099 611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:29.614795 611 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:36835
I20260620 23:16:29.614825 677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:36835 every 8 connection(s)
I20260620 23:16:29.616255 611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:16:29.620126 678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:29.623910 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 611
I20260620 23:16:29.624104 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:16:29.624542 11007 external_mini_cluster.cc:1428] Setting key 32449fe5d87f0a506dac9bdf83b87d6e
I20260620 23:16:29.633522 678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d0a0cb1bcc940739718c757c3dfce9c: Bootstrap starting.
I20260620 23:16:29.637019 678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d0a0cb1bcc940739718c757c3dfce9c: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:29.638092 678 log.cc:824] T 00000000000000000000000000000000 P 0d0a0cb1bcc940739718c757c3dfce9c: Log is configured to *not* fsync() on all Append() calls
Jun 20 23:16:29 dist-test-slave-jkbq krb5kdc[598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:16:29.641347 678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d0a0cb1bcc940739718c757c3dfce9c: No bootstrap required, opened a new log
I20260620 23:16:29.649395 681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:29.626487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59593 (local address 127.10.191.254:36835)
0620 23:16:29.627334 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:29.627350 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:29.627402 (+ 52us) server_negotiation.cc:407] Connection header received
0620 23:16:29.629203 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:29.629232 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:29.629798 (+ 566us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:29.630253 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:29.631654 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:29.633610 (+ 1956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:29.634914 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:29.635969 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:29.636182 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:29.640531 (+ 4349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:29.640542 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:29.640560 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:29.640603 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:29.644753 (+ 4150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:29.645415 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:29.645424 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:29.645440 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:29.645668 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:29.646036 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:29.646042 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:29.646045 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:29.646770 (+ 725us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:29.647163 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:29.647455 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 23:16:29.647930 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":378,"threads_started":1}
I20260620 23:16:29.652318 678 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d0a0cb1bcc940739718c757c3dfce9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d0a0cb1bcc940739718c757c3dfce9c" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 36835 } }
I20260620 23:16:29.652750 678 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d0a0cb1bcc940739718c757c3dfce9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:29.652897 678 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d0a0cb1bcc940739718c757c3dfce9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d0a0cb1bcc940739718c757c3dfce9c, State: Initialized, Role: FOLLOWER
I20260620 23:16:29.653501 11007 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:16:29.653533 678 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d0a0cb1bcc940739718c757c3dfce9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d0a0cb1bcc940739718c757c3dfce9c" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 36835 } }
I20260620 23:16:29.653846 678 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d0a0cb1bcc940739718c757c3dfce9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:16:29.653993 678 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d0a0cb1bcc940739718c757c3dfce9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:16:29.654177 678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d0a0cb1bcc940739718c757c3dfce9c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:29.656612 678 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d0a0cb1bcc940739718c757c3dfce9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d0a0cb1bcc940739718c757c3dfce9c" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 36835 } }
I20260620 23:16:29.657115 678 leader_election.cc:302] T 00000000000000000000000000000000 P 0d0a0cb1bcc940739718c757c3dfce9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d0a0cb1bcc940739718c757c3dfce9c; no voters:
I20260620 23:16:29.657734 678 leader_election.cc:288] T 00000000000000000000000000000000 P 0d0a0cb1bcc940739718c757c3dfce9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:16:29.657895 687 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d0a0cb1bcc940739718c757c3dfce9c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:16:29.660213 678 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d0a0cb1bcc940739718c757c3dfce9c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:16:29.660422 687 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d0a0cb1bcc940739718c757c3dfce9c [term 1 LEADER]: Becoming Leader. State: Replica: 0d0a0cb1bcc940739718c757c3dfce9c, State: Running, Role: LEADER
I20260620 23:16:29.660971 687 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d0a0cb1bcc940739718c757c3dfce9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d0a0cb1bcc940739718c757c3dfce9c" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 36835 } }
I20260620 23:16:29.664135 688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d0a0cb1bcc940739718c757c3dfce9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d0a0cb1bcc940739718c757c3dfce9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d0a0cb1bcc940739718c757c3dfce9c" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 36835 } } }
I20260620 23:16:29.664677 688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d0a0cb1bcc940739718c757c3dfce9c [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:29.664266 690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d0a0cb1bcc940739718c757c3dfce9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d0a0cb1bcc940739718c757c3dfce9c. Latest consensus state: current_term: 1 leader_uuid: "0d0a0cb1bcc940739718c757c3dfce9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d0a0cb1bcc940739718c757c3dfce9c" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 36835 } } }
I20260620 23:16:29.665280 690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d0a0cb1bcc940739718c757c3dfce9c [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:29.666533 696 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:16:29.672715 696 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:16:29.679389 681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:29.658107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53451 (local address 127.10.191.254:36835)
0620 23:16:29.658284 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:29.658292 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:29.659762 (+ 1470us) server_negotiation.cc:407] Connection header received
0620 23:16:29.659992 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:29.659997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:29.660080 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:29.660187 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:29.662478 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:29.664048 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:29.666436 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:29.667141 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:29.667285 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:29.670674 (+ 3389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:29.670682 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:29.670688 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:29.670733 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:29.675002 (+ 4269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:29.676187 (+ 1185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:29.676193 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:29.676196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:29.676359 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:29.676839 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:29.676846 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:29.676850 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:29.677354 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:29.677611 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:29.678757 (+ 1146us) server_negotiation.cc:299] Negotiation successful
0620 23:16:29.678997 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 23:16:29.684404 696 catalog_manager.cc:1347] Generated new cluster ID: ed3b3b64286548debc3228b04feb27e1
I20260620 23:16:29.684512 696 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:16:29.717721 696 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:16:29.719151 696 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:16:29.726428 696 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d0a0cb1bcc940739718c757c3dfce9c: Generated new TSK 0
I20260620 23:16:29.727277 696 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:16:29.733083 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 611
I20260620 23:16:29.734282 673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:16:29.824344 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 611
2026-06-20T23:16:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2023 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997042893763-11007-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:16:29 dist-test-slave-jkbq krb5kdc[710](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:16:29 dist-test-slave-jkbq krb5kdc[710](info): set up 2 sockets
Jun 20 23:16:29 dist-test-slave-jkbq krb5kdc[710](info): commencing operation
krb5kdc: starting...
W20260620 23:16:30.927390 11007 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Jun 20 23:16:31 dist-test-slave-jkbq krb5kdc[710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:16:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:16:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.191.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
I20260620 23:16:31.188380 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:45699
--webserver_interface=127.10.191.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:45699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:16:31.508975 723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:31.509255 723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:31.509423 723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:31.516510 723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:16:31.516606 723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:31.516710 723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:31.516820 723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:16:31.516901 723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:16:31.526615 723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:45699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:45699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:16:31.527848 723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:31.529157 723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:31.540482 731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:31.540570 728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:31.540510 729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:31.541821 723 server_base.cc:1034] running on GCE node
I20260620 23:16:31.542575 723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:31.543959 723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:31.545192 723 hybrid_clock.cc:648] HybridClock initialized: now 1781997391545153 us; error 51 us; skew 500 ppm
I20260620 23:16:31.545524 723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:16:31 dist-test-slave-jkbq krb5kdc[710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997391, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:31.548643 723 init.cc:377] Logged in from keytab as kudu/127.10.191.254@KRBTEST.COM (short username kudu)
I20260620 23:16:31.550135 723 webserver.cc:460] Webserver started at http://127.10.191.254:38541/ using document root <none> and password file <none>
I20260620 23:16:31.550801 723 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:31.550976 723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:31.551286 723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:31.554131 723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "b4bec9c366014386bcac5b4706a8b128"
format_stamp: "Formatted at 2026-06-20 23:16:31 on dist-test-slave-jkbq"
server_key: "100ceef48acd07c45142cfb2eb442a4c"
server_key_iv: "9948ae181c55ad21efd25d950505edde"
server_key_version: "encryptionkey@0"
I20260620 23:16:31.554946 723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "b4bec9c366014386bcac5b4706a8b128"
format_stamp: "Formatted at 2026-06-20 23:16:31 on dist-test-slave-jkbq"
server_key: "100ceef48acd07c45142cfb2eb442a4c"
server_key_iv: "9948ae181c55ad21efd25d950505edde"
server_key_version: "encryptionkey@0"
I20260620 23:16:31.560022 723 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 23:16:31.563546 738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:31.565063 723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:16:31.565285 723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "b4bec9c366014386bcac5b4706a8b128"
format_stamp: "Formatted at 2026-06-20 23:16:31 on dist-test-slave-jkbq"
server_key: "100ceef48acd07c45142cfb2eb442a4c"
server_key_iv: "9948ae181c55ad21efd25d950505edde"
server_key_version: "encryptionkey@0"
I20260620 23:16:31.565482 723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:31.588063 723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:31.592243 723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:31.592542 723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:31.611334 723 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:45699
I20260620 23:16:31.611354 789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:45699 every 8 connection(s)
I20260620 23:16:31.612701 723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:16:31.616357 790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:31.621757 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 723
I20260620 23:16:31.621932 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:16:31.622534 11007 external_mini_cluster.cc:1428] Setting key 3a26c4dea0e72dee7b68e598c16e0066
I20260620 23:16:31.627481 790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4bec9c366014386bcac5b4706a8b128: Bootstrap starting.
I20260620 23:16:31.630466 790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4bec9c366014386bcac5b4706a8b128: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:31.631522 790 log.cc:824] T 00000000000000000000000000000000 P b4bec9c366014386bcac5b4706a8b128: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:31.634686 790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4bec9c366014386bcac5b4706a8b128: No bootstrap required, opened a new log
I20260620 23:16:31.635722 793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:31.624542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56807 (local address 127.10.191.254:45699)
0620 23:16:31.625379 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:31.625393 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:31.625435 (+ 42us) server_negotiation.cc:407] Connection header received
0620 23:16:31.627172 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:31.627194 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:31.627625 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:31.628011 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:31.628531 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:31.629833 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:31.631254 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:31.632129 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:31.632334 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:31.633429 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:31.633436 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:31.633450 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:31.633970 (+ 520us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:31.634036 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:31.634060 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:31.634152 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:31.634385 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 23:16:31.634658 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":388,"threads_started":1}
I20260620 23:16:31.639585 11007 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:16:31.643128 790 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4bec9c366014386bcac5b4706a8b128 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4bec9c366014386bcac5b4706a8b128" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 45699 } }
I20260620 23:16:31.643514 790 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4bec9c366014386bcac5b4706a8b128 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:31.643632 790 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4bec9c366014386bcac5b4706a8b128 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4bec9c366014386bcac5b4706a8b128, State: Initialized, Role: FOLLOWER
I20260620 23:16:31.644277 790 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4bec9c366014386bcac5b4706a8b128 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4bec9c366014386bcac5b4706a8b128" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 45699 } }
I20260620 23:16:31.644524 790 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4bec9c366014386bcac5b4706a8b128 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:16:31.644670 790 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4bec9c366014386bcac5b4706a8b128 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:16:31.644855 790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4bec9c366014386bcac5b4706a8b128 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:31.647107 790 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4bec9c366014386bcac5b4706a8b128 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4bec9c366014386bcac5b4706a8b128" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 45699 } }
I20260620 23:16:31.647718 790 leader_election.cc:302] T 00000000000000000000000000000000 P b4bec9c366014386bcac5b4706a8b128 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4bec9c366014386bcac5b4706a8b128; no voters:
I20260620 23:16:31.648422 790 leader_election.cc:288] T 00000000000000000000000000000000 P b4bec9c366014386bcac5b4706a8b128 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:16:31.648535 800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4bec9c366014386bcac5b4706a8b128 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:16:31.649709 800 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4bec9c366014386bcac5b4706a8b128 [term 1 LEADER]: Becoming Leader. State: Replica: b4bec9c366014386bcac5b4706a8b128, State: Running, Role: LEADER
I20260620 23:16:31.650271 800 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4bec9c366014386bcac5b4706a8b128 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4bec9c366014386bcac5b4706a8b128" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 45699 } }
I20260620 23:16:31.650841 790 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4bec9c366014386bcac5b4706a8b128 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:16:31.653223 801 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4bec9c366014386bcac5b4706a8b128 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4bec9c366014386bcac5b4706a8b128" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4bec9c366014386bcac5b4706a8b128" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 45699 } } }
I20260620 23:16:31.653812 801 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4bec9c366014386bcac5b4706a8b128 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:31.655012 808 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:16:31.654899 802 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4bec9c366014386bcac5b4706a8b128 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4bec9c366014386bcac5b4706a8b128. Latest consensus state: current_term: 1 leader_uuid: "b4bec9c366014386bcac5b4706a8b128" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4bec9c366014386bcac5b4706a8b128" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 45699 } } }
I20260620 23:16:31.655375 802 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4bec9c366014386bcac5b4706a8b128 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:31.655460 793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:31.643890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46533 (local address 127.10.191.254:45699)
0620 23:16:31.644106 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:31.644115 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:31.647501 (+ 3386us) server_negotiation.cc:407] Connection header received
0620 23:16:31.647687 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:31.647691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:31.647756 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:31.647839 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:31.648518 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:31.649885 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:31.651783 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:31.652758 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:31.652903 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:31.654103 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:31.654111 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:31.654117 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:31.654145 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:31.654202 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:31.654232 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:31.654348 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:31.655010 (+ 662us) server_negotiation.cc:299] Negotiation successful
0620 23:16:31.655122 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:16:31.661893 808 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:16:31.670034 808 catalog_manager.cc:1347] Generated new cluster ID: d2a93ddef95446cb9a8ce8093c9239f8
I20260620 23:16:31.670145 808 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:16:31.684587 808 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:16:31.685874 808 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:16:31.706298 808 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4bec9c366014386bcac5b4706a8b128: Generated new TSK 0
I20260620 23:16:31.707262 808 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:16:31.723824 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 723
W20260620 23:16:31.725004 785 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56807
W20260620 23:16:31.725749 11007 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 723: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:16:31.725903 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 723
2026-06-20T23:16:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1900 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997042893763-11007-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:16:31 dist-test-slave-jkbq krb5kdc[821](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:16:31 dist-test-slave-jkbq krb5kdc[821](info): set up 2 sockets
Jun 20 23:16:31 dist-test-slave-jkbq krb5kdc[821](info): commencing operation
krb5kdc: starting...
W20260620 23:16:32.831487 11007 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Jun 20 23:16:32 dist-test-slave-jkbq krb5kdc[821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:16:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:16:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.191.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
I20260620 23:16:33.074813 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:43315
--webserver_interface=127.10.191.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:40219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:43315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:16:33.395278 834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:33.395594 834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:33.395767 834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:33.402822 834 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:16:33.402988 834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:33.403052 834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:33.403115 834 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:16:33.403157 834 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:16:33.412670 834 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:40219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:43315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:43315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:16:33.413839 834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:33.415107 834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:33.426734 842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:33.426769 840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:33.426689 839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:33.427477 834 server_base.cc:1034] running on GCE node
I20260620 23:16:33.428123 834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:33.429478 834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:33.430675 834 hybrid_clock.cc:648] HybridClock initialized: now 1781997393430645 us; error 29 us; skew 500 ppm
I20260620 23:16:33.431030 834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:16:33 dist-test-slave-jkbq krb5kdc[821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997393, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:33.438095 834 init.cc:377] Logged in from keytab as kudu/127.10.191.254@KRBTEST.COM (short username kudu)
I20260620 23:16:33.439572 834 webserver.cc:460] Webserver started at http://127.10.191.254:45505/ using document root <none> and password file <none>
I20260620 23:16:33.440209 834 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:33.440316 834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:33.440549 834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:33.443444 834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "283ffb6008134d73a8cc28de3229eb66"
format_stamp: "Formatted at 2026-06-20 23:16:33 on dist-test-slave-jkbq"
server_key: "bd1908352e0890be1cc652d158ebb6ce"
server_key_iv: "d4858ea6b0fedb801cace6020193953e"
server_key_version: "encryptionkey@0"
I20260620 23:16:33.444136 834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "283ffb6008134d73a8cc28de3229eb66"
format_stamp: "Formatted at 2026-06-20 23:16:33 on dist-test-slave-jkbq"
server_key: "bd1908352e0890be1cc652d158ebb6ce"
server_key_iv: "d4858ea6b0fedb801cace6020193953e"
server_key_version: "encryptionkey@0"
I20260620 23:16:33.448971 834 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:16:33.452291 849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:33.453550 834 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260620 23:16:33.453678 834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "283ffb6008134d73a8cc28de3229eb66"
format_stamp: "Formatted at 2026-06-20 23:16:33 on dist-test-slave-jkbq"
server_key: "bd1908352e0890be1cc652d158ebb6ce"
server_key_iv: "d4858ea6b0fedb801cace6020193953e"
server_key_version: "encryptionkey@0"
I20260620 23:16:33.453850 834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:33.459436 834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:33.463631 834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:33.463868 834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:33.481666 834 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:43315
I20260620 23:16:33.481706 900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:43315 every 8 connection(s)
I20260620 23:16:33.483047 834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:16:33.486550 901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:33.487574 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 834
I20260620 23:16:33.487756 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:16:33.488303 11007 external_mini_cluster.cc:1428] Setting key 9733221f0422ba9436ec78fb72c19ce4
I20260620 23:16:33.495894 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:33.490028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51289 (local address 127.10.191.254:43315)
0620 23:16:33.490803 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:33.490814 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:33.490856 (+ 42us) server_negotiation.cc:407] Connection header received
0620 23:16:33.492722 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:33.492748 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:33.493081 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:33.493410 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:33.494404 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:33.494409 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:33.494422 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:33.494435 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:33.494489 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:33.494510 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:33.494666 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:33.494791 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 23:16:33.495013 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":355,"threads_started":1}
I20260620 23:16:33.499413 11007 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:16:33.499575 901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 283ffb6008134d73a8cc28de3229eb66: Bootstrap starting.
I20260620 23:16:33.503311 901 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 283ffb6008134d73a8cc28de3229eb66: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:33.504586 901 log.cc:824] T 00000000000000000000000000000000 P 283ffb6008134d73a8cc28de3229eb66: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:33.507161 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:33.504593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37595 (local address 127.10.191.254:43315)
0620 23:16:33.504775 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:33.504781 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:33.505219 (+ 438us) server_negotiation.cc:407] Connection header received
0620 23:16:33.505376 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:33.505380 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:33.505421 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:33.505500 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:33.506509 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:33.506514 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:33.506518 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:16:33.506538 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:33.506582 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:16:33.506602 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:33.506714 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:33.506793 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 23:16:33.506848 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 23:16:33.507741 901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 283ffb6008134d73a8cc28de3229eb66: No bootstrap required, opened a new log
I20260620 23:16:33.514626 901 raft_consensus.cc:348] T 00000000000000000000000000000000 P 283ffb6008134d73a8cc28de3229eb66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "283ffb6008134d73a8cc28de3229eb66" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43315 } }
I20260620 23:16:33.515035 901 raft_consensus.cc:374] T 00000000000000000000000000000000 P 283ffb6008134d73a8cc28de3229eb66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:16:33.515144 901 raft_consensus.cc:729] T 00000000000000000000000000000000 P 283ffb6008134d73a8cc28de3229eb66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 283ffb6008134d73a8cc28de3229eb66, State: Initialized, Role: FOLLOWER
I20260620 23:16:33.515743 901 consensus_queue.cc:260] T 00000000000000000000000000000000 P 283ffb6008134d73a8cc28de3229eb66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "283ffb6008134d73a8cc28de3229eb66" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43315 } }
I20260620 23:16:33.515970 901 raft_consensus.cc:388] T 00000000000000000000000000000000 P 283ffb6008134d73a8cc28de3229eb66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:16:33.516101 901 raft_consensus.cc:482] T 00000000000000000000000000000000 P 283ffb6008134d73a8cc28de3229eb66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:16:33.516266 901 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 283ffb6008134d73a8cc28de3229eb66 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:33.518333 901 raft_consensus.cc:504] T 00000000000000000000000000000000 P 283ffb6008134d73a8cc28de3229eb66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "283ffb6008134d73a8cc28de3229eb66" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43315 } }
I20260620 23:16:33.518793 901 leader_election.cc:302] T 00000000000000000000000000000000 P 283ffb6008134d73a8cc28de3229eb66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 283ffb6008134d73a8cc28de3229eb66; no voters:
I20260620 23:16:33.519405 901 leader_election.cc:288] T 00000000000000000000000000000000 P 283ffb6008134d73a8cc28de3229eb66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:16:33.519541 911 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 283ffb6008134d73a8cc28de3229eb66 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:16:33.520234 911 raft_consensus.cc:686] T 00000000000000000000000000000000 P 283ffb6008134d73a8cc28de3229eb66 [term 1 LEADER]: Becoming Leader. State: Replica: 283ffb6008134d73a8cc28de3229eb66, State: Running, Role: LEADER
I20260620 23:16:33.520789 911 consensus_queue.cc:237] T 00000000000000000000000000000000 P 283ffb6008134d73a8cc28de3229eb66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "283ffb6008134d73a8cc28de3229eb66" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43315 } }
I20260620 23:16:33.521329 901 sys_catalog.cc:564] T 00000000000000000000000000000000 P 283ffb6008134d73a8cc28de3229eb66 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:16:33.523648 913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 283ffb6008134d73a8cc28de3229eb66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 283ffb6008134d73a8cc28de3229eb66. Latest consensus state: current_term: 1 leader_uuid: "283ffb6008134d73a8cc28de3229eb66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "283ffb6008134d73a8cc28de3229eb66" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43315 } } }
I20260620 23:16:33.523756 912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 283ffb6008134d73a8cc28de3229eb66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "283ffb6008134d73a8cc28de3229eb66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "283ffb6008134d73a8cc28de3229eb66" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 43315 } } }
I20260620 23:16:33.524080 913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 283ffb6008134d73a8cc28de3229eb66 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:33.524135 912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 283ffb6008134d73a8cc28de3229eb66 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:33.524982 918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:16:33.530589 918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:16:33.538810 918 catalog_manager.cc:1347] Generated new cluster ID: ec5877d56bbe4d65ba603173759254ea
I20260620 23:16:33.538987 918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:16:33.568830 918 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:16:33.569259 918 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:16:33.579047 918 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 283ffb6008134d73a8cc28de3229eb66: Generated new TSK 0
I20260620 23:16:33.580003 918 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:16:33.596560 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 834
W20260620 23:16:33.597631 896 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51289
W20260620 23:16:33.598349 11007 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 834: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:16:33.598496 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 834
2026-06-20T23:16:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1872 ms)
[----------] 18 tests from AuthTokenIssuingTest (34511 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:16:33 dist-test-slave-jkbq krb5kdc[932](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:16:33 dist-test-slave-jkbq krb5kdc[932](info): set up 2 sockets
Jun 20 23:16:33 dist-test-slave-jkbq krb5kdc[932](info): commencing operation
krb5kdc: starting...
W20260620 23:16:34.695513 11007 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Jun 20 23:16:34 dist-test-slave-jkbq krb5kdc[932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:16:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:16:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.191.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
I20260620 23:16:34.946424 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:33619
--webserver_interface=127.10.191.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:46283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:33619,127.10.191.253:46095,127.10.191.252:39993
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:16:35.268019 945 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:16:35.268411 945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:35.268540 945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:35.268680 945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:35.275735 945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:16:35.275861 945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:35.275969 945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:35.276055 945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:16:35.276144 945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:16:35.285861 945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:46283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:33619,127.10.191.253:46095,127.10.191.252:39993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:33619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:16:35.287182 945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:35.288476 945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:35.300292 953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:35.300331 950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:35.300370 951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:35.301465 945 server_base.cc:1034] running on GCE node
I20260620 23:16:35.302263 945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:35.303754 945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:35.305004 945 hybrid_clock.cc:648] HybridClock initialized: now 1781997395304952 us; error 37 us; skew 500 ppm
I20260620 23:16:35.305367 945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:16:35 dist-test-slave-jkbq krb5kdc[932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997395, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:35.308568 945 init.cc:377] Logged in from keytab as kudu/127.10.191.254@KRBTEST.COM (short username kudu)
I20260620 23:16:35.310051 945 webserver.cc:460] Webserver started at http://127.10.191.254:37865/ using document root <none> and password file <none>
I20260620 23:16:35.310724 945 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:35.310853 945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:35.311237 945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:35.314101 945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "b768f9fc187c4c33814a4f02b55689e6"
format_stamp: "Formatted at 2026-06-20 23:16:35 on dist-test-slave-jkbq"
server_key: "50d11fddab2eae0e88651daba2bf9cb5"
server_key_iv: "927a94f116ee4a3f0e84436341a45bd8"
server_key_version: "encryptionkey@0"
I20260620 23:16:35.314955 945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "b768f9fc187c4c33814a4f02b55689e6"
format_stamp: "Formatted at 2026-06-20 23:16:35 on dist-test-slave-jkbq"
server_key: "50d11fddab2eae0e88651daba2bf9cb5"
server_key_iv: "927a94f116ee4a3f0e84436341a45bd8"
server_key_version: "encryptionkey@0"
I20260620 23:16:35.319887 945 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 23:16:35.323237 960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:35.324476 945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:16:35.324646 945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "b768f9fc187c4c33814a4f02b55689e6"
format_stamp: "Formatted at 2026-06-20 23:16:35 on dist-test-slave-jkbq"
server_key: "50d11fddab2eae0e88651daba2bf9cb5"
server_key_iv: "927a94f116ee4a3f0e84436341a45bd8"
server_key_version: "encryptionkey@0"
I20260620 23:16:35.324851 945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:35.352716 945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:35.357071 945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:35.357368 945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:35.375922 945 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:33619
I20260620 23:16:35.375945 1011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:33619 every 8 connection(s)
I20260620 23:16:35.377230 945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:16:35.380764 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 945
I20260620 23:16:35.380985 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:16:35.381109 1012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:35.381829 11007 external_mini_cluster.cc:1428] Setting key 7afb35f781048424a24f37818895b69f
I20260620 23:16:35.390933 1012 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.191.254" port: 33619 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:16:35 dist-test-slave-jkbq krb5kdc[932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997395, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.191.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.253@KRBTEST.COM" created.
I20260620 23:16:35.413288 1016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:35.394525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:33619 (local address 127.0.0.1:33282)
0620 23:16:35.395706 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:35.395737 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:35.397432 (+ 1695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:35.398236 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:35.398243 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:35.399004 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:16:35.399333 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:35.399349 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:35.400867 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:35.400872 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:35.401920 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:35.401931 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:35.402833 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:35.402839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:35.403044 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:35.403710 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:16:35.403731 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:16:35.406076 (+ 2345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:16:35.409174 (+ 3098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:35.409184 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:35.409191 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:35.409866 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:35.410353 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:35.410356 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:35.410359 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:35.410500 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:35.411495 (+ 995us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:16:35.411500 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:16:35.411703 (+ 203us) client_negotiation.cc:769] Sending connection context
0620 23:16:35.411909 (+ 206us) client_negotiation.cc:240] Negotiation successful
0620 23:16:35.412277 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1005,"thread_start_us":848,"threads_started":1}
I20260620 23:16:35.413326 1015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:35.394515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33282 (local address 127.10.191.254:33619)
0620 23:16:35.395314 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:35.395324 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:35.395853 (+ 529us) server_negotiation.cc:407] Connection header received
0620 23:16:35.397791 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:35.397797 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:35.398010 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:35.398209 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:35.399541 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:35.400712 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:35.402057 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:35.402603 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:35.402806 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:35.406297 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:35.406305 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:35.406316 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:35.406343 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:35.409050 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:35.410027 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:35.410036 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:35.410041 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:35.410238 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:35.410616 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:35.410625 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:35.410631 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:35.411358 (+ 727us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:35.411572 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:35.412110 (+ 538us) server_negotiation.cc:299] Negotiation successful
0620 23:16:35.412471 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":602,"thread_start_us":395,"threads_started":1}
I20260620 23:16:35.416230 1012 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.191.253" port: 46095 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:16:35.417330 1016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:35.416846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:46095 (local address 127.0.0.1:55620)
0620 23:16:35.417043 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:35.417176 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.253:46095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 23:16:35.417600 964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.191.253:46095: connect: Connection refused (error 111)
I20260620 23:16:35.419163 1016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:35.418774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:46095 (local address 127.0.0.1:55634)
0620 23:16:35.418963 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:35.419039 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.253:46095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 23:16:35.419438 1012 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.191.253:46095: Network error: Client connection negotiation failed: client connection to 127.10.191.253:46095: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.253.keytab.
Entry for principal kudu/127.10.191.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.253.keytab.
I20260620 23:16:35.438546 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.253:46095
--webserver_interface=127.10.191.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.253.keytab
--principal=kudu/127.10.191.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:46283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:33619,127.10.191.253:46095,127.10.191.252:39993
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:16:35.468786 1012 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.191.253" port: 46095 } attempt: 1
I20260620 23:16:35.470024 1016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:35.469549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:46095 (local address 127.0.0.1:55642)
0620 23:16:35.469766 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:35.469873 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.253:46095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 23:16:35.471096 1016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:35.470702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:46095 (local address 127.0.0.1:55650)
0620 23:16:35.470860 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:35.470965 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.253:46095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 23:16:35.471277 1012 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.191.253:46095: Network error: Client connection negotiation failed: client connection to 127.10.191.253:46095: connect: Connection refused (error 111)
I20260620 23:16:35.539525 1012 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.191.253" port: 46095 } attempt: 2
I20260620 23:16:35.540720 1016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:35.540215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:46095 (local address 127.0.0.1:55656)
0620 23:16:35.540483 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:35.540574 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.253:46095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 23:16:35.541857 1016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:35.541485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:46095 (local address 127.0.0.1:55666)
0620 23:16:35.541655 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:35.541735 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.253:46095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 23:16:35.542035 1012 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.191.253:46095: Network error: Client connection negotiation failed: client connection to 127.10.191.253:46095: connect: Connection refused (error 111)
I20260620 23:16:35.633323 1012 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.191.253" port: 46095 } attempt: 3
I20260620 23:16:35.634478 1016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:35.634049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:46095 (local address 127.0.0.1:55678)
0620 23:16:35.634246 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:35.634343 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.253:46095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 23:16:35.635586 1016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:35.635225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:46095 (local address 127.0.0.1:55680)
0620 23:16:35.635380 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:35.635457 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.253:46095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 23:16:35.635774 1012 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.191.253:46095: Network error: Client connection negotiation failed: client connection to 127.10.191.253:46095: connect: Connection refused (error 111)
W20260620 23:16:35.755302 1019 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:16:35.755678 1019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:35.755803 1019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:35.755941 1019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:35.763167 1019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:16:35.763299 1019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:35.763427 1019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:35.763513 1019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:16:35.763605 1019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:16:35.773193 1019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:46283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:33619,127.10.191.253:46095,127.10.191.252:39993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.253.keytab
--principal=kudu/127.10.191.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.253:46095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.191.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:16:35.774477 1019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:35.775852 1019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 23:16:35.779039 1012 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.191.253" port: 46095 } attempt: 4
I20260620 23:16:35.780193 1016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:35.779674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:46095 (local address 127.0.0.1:55692)
0620 23:16:35.779974 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:35.780057 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.253:46095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260620 23:16:35.781400 1016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:35.780924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:46095 (local address 127.0.0.1:55696)
0620 23:16:35.781116 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:35.781222 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.253:46095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260620 23:16:35.781769 1012 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.191.253:46095: Network error: Client connection negotiation failed: client connection to 127.10.191.253:46095: connect: Connection refused (error 111)
W20260620 23:16:35.787573 1027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:35.787611 1025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:35.787531 1024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:35.788599 1019 server_base.cc:1034] running on GCE node
I20260620 23:16:35.789376 1019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:35.790759 1019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:35.792002 1019 hybrid_clock.cc:648] HybridClock initialized: now 1781997395791943 us; error 35 us; skew 500 ppm
I20260620 23:16:35.792348 1019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:16:35 dist-test-slave-jkbq krb5kdc[932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997395, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:35.795555 1019 init.cc:377] Logged in from keytab as kudu/127.10.191.253@KRBTEST.COM (short username kudu)
I20260620 23:16:35.797019 1019 webserver.cc:460] Webserver started at http://127.10.191.253:38557/ using document root <none> and password file <none>
I20260620 23:16:35.797910 1019 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:35.798040 1019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:35.798485 1019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:35.801458 1019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-1/data/instance:
uuid: "41129b9f87b94019b56b19ad796771f1"
format_stamp: "Formatted at 2026-06-20 23:16:35 on dist-test-slave-jkbq"
server_key: "fb40e4da83b5277a5020ba447b8cfd7e"
server_key_iv: "8ca8f394c375a6f2760716b52928fcc0"
server_key_version: "encryptionkey@0"
I20260620 23:16:35.802191 1019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-1/wal/instance:
uuid: "41129b9f87b94019b56b19ad796771f1"
format_stamp: "Formatted at 2026-06-20 23:16:35 on dist-test-slave-jkbq"
server_key: "fb40e4da83b5277a5020ba447b8cfd7e"
server_key_iv: "8ca8f394c375a6f2760716b52928fcc0"
server_key_version: "encryptionkey@0"
I20260620 23:16:35.807169 1019 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 23:16:35.810547 1034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:35.811926 1019 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:16:35.812059 1019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-1/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-1/wal
uuid: "41129b9f87b94019b56b19ad796771f1"
format_stamp: "Formatted at 2026-06-20 23:16:35 on dist-test-slave-jkbq"
server_key: "fb40e4da83b5277a5020ba447b8cfd7e"
server_key_iv: "8ca8f394c375a6f2760716b52928fcc0"
server_key_version: "encryptionkey@0"
I20260620 23:16:35.812229 1019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:35.841985 1019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:35.846393 1019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:35.846621 1019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:35.866117 1019 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.253:46095
I20260620 23:16:35.866165 1085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.253:46095 every 8 connection(s)
I20260620 23:16:35.867568 1019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-1/data/info.pb
I20260620 23:16:35.870968 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 1019
I20260620 23:16:35.871232 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-1/wal/instance
I20260620 23:16:35.871546 1086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:35.871695 11007 external_mini_cluster.cc:1428] Setting key d16acef0a99f0d507a0a906e51a6d754
I20260620 23:16:35.878123 1086 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.191.254" port: 33619 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:16:35 dist-test-slave-jkbq krb5kdc[932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997395, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.253@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.191.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.252@KRBTEST.COM" created.
I20260620 23:16:35.898527 1015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:35.880805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33296 (local address 127.10.191.254:33619)
0620 23:16:35.880992 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:35.880996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:35.882242 (+ 1246us) server_negotiation.cc:407] Connection header received
0620 23:16:35.884172 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:35.884177 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:35.884275 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:35.884387 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:35.885875 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:35.886563 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:35.888136 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:35.888539 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:35.888704 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:35.892386 (+ 3682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:35.892396 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:35.892402 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:35.892451 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:35.895498 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:35.896289 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:35.896293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:35.896296 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:35.896422 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:35.896847 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:35.896851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:35.896853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:35.897118 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:35.897303 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:35.897969 (+ 666us) server_negotiation.cc:299] Negotiation successful
0620 23:16:35.898187 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 23:16:35.899560 1089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:35.881329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:33619 (local address 127.0.0.1:33296)
0620 23:16:35.882066 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:35.882117 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:35.883899 (+ 1782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:35.884624 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:35.884634 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:35.885144 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:16:35.885704 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:35.885721 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:35.886739 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:35.886743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:35.887981 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:35.887993 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:35.888808 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:35.888815 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:35.889040 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:35.889732 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:16:35.889754 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:16:35.892140 (+ 2386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:16:35.895620 (+ 3480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:35.895627 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:35.895639 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:35.896142 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:35.896534 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:35.896539 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:35.896543 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:35.896721 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:35.897253 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:16:35.897261 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:16:35.897600 (+ 339us) client_negotiation.cc:769] Sending connection context
0620 23:16:35.897866 (+ 266us) client_negotiation.cc:240] Negotiation successful
0620 23:16:35.898372 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":320,"threads_started":1}
I20260620 23:16:35.901355 1086 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.191.253" port: 46095 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:16:35 dist-test-slave-jkbq krb5kdc[932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997395, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.253@KRBTEST.COM for kudu/127.10.191.253@KRBTEST.COM
I20260620 23:16:35.916100 1090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:35.902270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55706 (local address 127.10.191.253:46095)
0620 23:16:35.902780 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:35.902785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:35.902809 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:16:35.902848 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:35.902853 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:35.903084 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:35.903253 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:35.903964 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:35.904983 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:35.906077 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:35.906598 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:35.906730 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:35.910089 (+ 3359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:35.910099 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:35.910112 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:35.910138 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:35.912882 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:35.913742 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:35.913748 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:35.913750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:35.913857 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:35.914282 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:35.914285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:35.914287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:35.914607 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:35.914768 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:35.915450 (+ 682us) server_negotiation.cc:299] Negotiation successful
0620 23:16:35.915797 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":102,"server-negotiator.queue_time_us":379,"thread_start_us":240,"threads_started":1}
I20260620 23:16:35.916182 1089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:35.901997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:46095 (local address 127.0.0.1:55706)
0620 23:16:35.902155 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:35.902169 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:35.902250 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:35.903225 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:35.903229 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:35.903653 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:16:35.903779 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:35.903787 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:35.905114 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:35.905121 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:35.905947 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:35.905957 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:35.906744 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:35.906749 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:35.906963 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:35.907570 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:16:35.907597 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:16:35.909887 (+ 2290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:16:35.913026 (+ 3139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:35.913035 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:35.913042 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:35.913608 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:35.913967 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:35.913976 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:35.913982 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:35.914180 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:35.914930 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:16:35.914939 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:16:35.915124 (+ 185us) client_negotiation.cc:769] Sending connection context
0620 23:16:35.915291 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 23:16:35.915647 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":3}
I20260620 23:16:35.919034 1086 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.191.252" port: 39993 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:16:35.920148 1089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:35.919736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:39993 (local address 127.0.0.1:57026)
0620 23:16:35.919914 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:35.920010 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:39993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 23:16:35.920490 1037 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.191.252:39993: connect: Connection refused (error 111)
I20260620 23:16:35.922214 1089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:35.921797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:39993 (local address 127.0.0.1:57034)
0620 23:16:35.921990 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:35.922072 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:39993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 23:16:35.922545 1086 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.191.252:39993: Network error: Client connection negotiation failed: client connection to 127.10.191.252:39993: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.252.keytab.
Entry for principal kudu/127.10.191.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.252.keytab.
I20260620 23:16:35.928429 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.252:39993
--webserver_interface=127.10.191.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.252.keytab
--principal=kudu/127.10.191.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:46283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:33619,127.10.191.253:46095,127.10.191.252:39993
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:16:35.971872 1086 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.191.252" port: 39993 } attempt: 1
I20260620 23:16:35.973415 1089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:35.972735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:39993 (local address 127.0.0.1:57048)
0620 23:16:35.973043 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:35.973198 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:39993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 23:16:35.974392 1089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:35.974036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:39993 (local address 127.0.0.1:57060)
0620 23:16:35.974205 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:35.974292 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:39993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 23:16:35.974619 1086 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.191.252:39993: Network error: Client connection negotiation failed: client connection to 127.10.191.252:39993: connect: Connection refused (error 111)
I20260620 23:16:36.042838 1086 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.191.252" port: 39993 } attempt: 2
I20260620 23:16:36.044106 1089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.043550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:39993 (local address 127.0.0.1:57074)
0620 23:16:36.043790 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:36.043908 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:39993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 23:16:36.045181 1089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.044788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:39993 (local address 127.0.0.1:57078)
0620 23:16:36.044945 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:36.045032 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:39993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 23:16:36.045437 1086 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.191.252:39993: Network error: Client connection negotiation failed: client connection to 127.10.191.252:39993: connect: Connection refused (error 111)
I20260620 23:16:36.081041 1012 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.191.253" port: 46095 } attempt: 5
Jun 20 23:16:36 dist-test-slave-jkbq krb5kdc[932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997395, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for kudu/127.10.191.253@KRBTEST.COM
I20260620 23:16:36.096442 1016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.081902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:46095 (local address 127.0.0.1:55712)
0620 23:16:36.082202 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:36.082229 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:36.082359 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:36.082918 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:36.082923 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:36.083336 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:16:36.083517 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:36.083526 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.085181 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.085185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:36.086047 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:36.086058 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.087167 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.087172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:36.087410 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:36.088062 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:16:36.088079 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:16:36.090384 (+ 2305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:16:36.094117 (+ 3733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:36.094121 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:36.094123 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:36.094494 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:36.094965 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:36.094968 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:36.094971 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:36.095100 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:36.095724 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:16:36.095733 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:16:36.095856 (+ 123us) client_negotiation.cc:769] Sending connection context
0620 23:16:36.095975 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 23:16:36.096192 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 23:16:36.096652 1090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.082146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55712 (local address 127.10.191.253:46095)
0620 23:16:36.082387 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:36.082395 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:36.082450 (+ 55us) server_negotiation.cc:407] Connection header received
0620 23:16:36.082546 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:36.082554 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:36.082675 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:36.082803 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:36.083709 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.084982 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.086226 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.086990 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.087122 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:36.090568 (+ 3446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:36.090575 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:36.090580 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:36.090619 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:36.093973 (+ 3354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:36.094613 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:36.094618 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:36.094629 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:36.094788 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:36.095229 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:36.095234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:36.095238 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:36.095596 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:36.095759 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:36.096109 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 23:16:36.096337 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 23:16:36.097709 1012 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.191.252" port: 39993 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:16:36.098738 1016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.098346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:39993 (local address 127.0.0.1:57094)
0620 23:16:36.098530 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:36.098613 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:39993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 23:16:36.099833 1016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.099483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:39993 (local address 127.0.0.1:57104)
0620 23:16:36.099651 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:36.099728 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:39993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 23:16:36.100091 1012 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.191.252:39993: Network error: Client connection negotiation failed: client connection to 127.10.191.252:39993: connect: Connection refused (error 111)
I20260620 23:16:36.136684 1086 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.191.252" port: 39993 } attempt: 3
I20260620 23:16:36.138001 1089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.137401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:39993 (local address 127.0.0.1:57116)
0620 23:16:36.137651 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:36.137796 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:39993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 23:16:36.139249 1089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.138739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:39993 (local address 127.0.0.1:57132)
0620 23:16:36.139007 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:36.139105 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:39993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
W20260620 23:16:36.139492 1086 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.191.252:39993: Network error: Client connection negotiation failed: client connection to 127.10.191.252:39993: connect: Connection refused (error 111)
I20260620 23:16:36.151353 1012 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.191.252" port: 39993 } attempt: 1
I20260620 23:16:36.152472 1016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.152016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:39993 (local address 127.0.0.1:57144)
0620 23:16:36.152211 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:36.152307 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:39993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 23:16:36.153582 1016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.153233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:39993 (local address 127.0.0.1:57146)
0620 23:16:36.153416 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:36.153482 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:39993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 23:16:36.153808 1012 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.191.252:39993: Network error: Client connection negotiation failed: client connection to 127.10.191.252:39993: connect: Connection refused (error 111)
I20260620 23:16:36.222090 1012 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.191.252" port: 39993 } attempt: 2
I20260620 23:16:36.223326 1016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.222791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:39993 (local address 127.0.0.1:57148)
0620 23:16:36.223060 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:36.223159 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:39993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260620 23:16:36.224404 1016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.224039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:39993 (local address 127.0.0.1:57164)
0620 23:16:36.224201 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:36.224296 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:39993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260620 23:16:36.224684 1012 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.191.252:39993: Network error: Client connection negotiation failed: client connection to 127.10.191.252:39993: connect: Connection refused (error 111)
W20260620 23:16:36.257578 1093 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:16:36.257892 1093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:36.257990 1093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:36.258148 1093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:36.265285 1093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:16:36.265369 1093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:36.265467 1093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:36.265570 1093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:16:36.265681 1093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:16:36.276000 1093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:46283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:33619,127.10.191.253:46095,127.10.191.252:39993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.252.keytab
--principal=kudu/127.10.191.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.252:39993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.191.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:16:36.277302 1093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:36.278613 1093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 23:16:36.282696 1086 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.191.252" port: 39993 } attempt: 4
I20260620 23:16:36.284196 1089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.283555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:39993 (local address 127.0.0.1:57180)
0620 23:16:36.283815 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:36.283972 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:39993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 23:16:36.285570 1089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.285071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:39993 (local address 127.0.0.1:57192)
0620 23:16:36.285273 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:36.285388 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:39993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260620 23:16:36.285878 1086 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.191.252:39993: Network error: Client connection negotiation failed: client connection to 127.10.191.252:39993: connect: Connection refused (error 111)
W20260620 23:16:36.292527 1099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:36.292838 1098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:36.293177 1101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:36.294409 1093 server_base.cc:1034] running on GCE node
I20260620 23:16:36.295361 1093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:36.296720 1093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:36.297976 1093 hybrid_clock.cc:648] HybridClock initialized: now 1781997396297916 us; error 33 us; skew 500 ppm
I20260620 23:16:36.298316 1093 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:16:36 dist-test-slave-jkbq krb5kdc[932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997396, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:36.301601 1093 init.cc:377] Logged in from keytab as kudu/127.10.191.252@KRBTEST.COM (short username kudu)
I20260620 23:16:36.303061 1093 webserver.cc:460] Webserver started at http://127.10.191.252:38347/ using document root <none> and password file <none>
I20260620 23:16:36.303791 1093 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:36.303925 1093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:36.304253 1093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:36.307262 1093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-2/data/instance:
uuid: "73fe2761fc1a4204a26373cbc45abd1e"
format_stamp: "Formatted at 2026-06-20 23:16:36 on dist-test-slave-jkbq"
server_key: "f2d5e419745e40fd6ff678ab37cff4fb"
server_key_iv: "917d073a5d0120f8a91cbb1e85d68281"
server_key_version: "encryptionkey@0"
I20260620 23:16:36.308059 1093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-2/wal/instance:
uuid: "73fe2761fc1a4204a26373cbc45abd1e"
format_stamp: "Formatted at 2026-06-20 23:16:36 on dist-test-slave-jkbq"
server_key: "f2d5e419745e40fd6ff678ab37cff4fb"
server_key_iv: "917d073a5d0120f8a91cbb1e85d68281"
server_key_version: "encryptionkey@0"
I20260620 23:16:36.313246 1093 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260620 23:16:36.316843 1108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:36.318203 1093 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:16:36.318403 1093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-2/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-2/wal
uuid: "73fe2761fc1a4204a26373cbc45abd1e"
format_stamp: "Formatted at 2026-06-20 23:16:36 on dist-test-slave-jkbq"
server_key: "f2d5e419745e40fd6ff678ab37cff4fb"
server_key_iv: "917d073a5d0120f8a91cbb1e85d68281"
server_key_version: "encryptionkey@0"
I20260620 23:16:36.318622 1093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:36.330946 1012 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.191.252" port: 39993 } attempt: 3
I20260620 23:16:36.331938 1016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.331502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:39993 (local address 127.0.0.1:57200)
0620 23:16:36.331685 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:36.331783 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:39993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 23:16:36.332926 1016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.332528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:39993 (local address 127.0.0.1:57214)
0620 23:16:36.332740 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:36.332832 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:39993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260620 23:16:36.333128 1012 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.191.252:39993: Network error: Client connection negotiation failed: client connection to 127.10.191.252:39993: connect: Connection refused (error 111)
I20260620 23:16:36.337324 1093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:36.340997 1093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:36.341257 1093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:36.359539 1093 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.252:39993
I20260620 23:16:36.359551 1159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.252:39993 every 8 connection(s)
I20260620 23:16:36.360925 1093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-2/data/info.pb
I20260620 23:16:36.362169 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 1093
I20260620 23:16:36.362363 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781997042893763-11007-0/minicluster-data/master-2/wal/instance
I20260620 23:16:36.362882 11007 external_mini_cluster.cc:1428] Setting key d8ffce335e746ad745dc52811de5ded1
I20260620 23:16:36.365643 1160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:36.372408 1160 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.191.254" port: 33619 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:16:36 dist-test-slave-jkbq krb5kdc[932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.252@KRBTEST.COM
Jun 20 23:16:36 dist-test-slave-jkbq krb5kdc[932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997396, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.252@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:16:36.386620 1163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.364805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57222 (local address 127.10.191.252:39993)
0620 23:16:36.365719 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:36.365737 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:36.365792 (+ 55us) server_negotiation.cc:407] Connection header received
0620 23:16:36.367484 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:36.367505 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:36.367867 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:36.368294 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:36.369772 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.371153 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.372421 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.373097 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.373309 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:36.378305 (+ 4996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:36.378315 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:36.378327 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:36.378370 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:36.381844 (+ 3474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:36.382887 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:36.382896 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:36.382905 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:36.383109 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:36.383573 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:36.383579 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:36.383585 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:36.384252 (+ 667us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:36.384530 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:36.385021 (+ 491us) server_negotiation.cc:299] Negotiation successful
0620 23:16:36.385435 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":397,"threads_started":1}
I20260620 23:16:36.390987 1015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.375011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33300 (local address 127.10.191.254:33619)
0620 23:16:36.375195 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:36.375202 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:36.376031 (+ 829us) server_negotiation.cc:407] Connection header received
0620 23:16:36.376322 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:36.376331 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:36.376430 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:36.376541 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:36.377632 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.378492 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.380518 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.381199 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.381348 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:36.385005 (+ 3657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:36.385012 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:36.385019 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:36.385068 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:36.388080 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:36.388764 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:36.388768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:36.388770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:36.388940 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:36.389330 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:36.389335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:36.389338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:36.389694 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:36.389853 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:36.390360 (+ 507us) server_negotiation.cc:299] Negotiation successful
0620 23:16:36.390573 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 23:16:36.390997 1164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.375101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:33619 (local address 127.0.0.1:33300)
0620 23:16:36.375891 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:36.375940 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:36.376153 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:36.376600 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:36.376607 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:36.377244 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:16:36.377489 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:36.377497 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.378663 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.378669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:36.380362 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:36.380379 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.381403 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.381409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:36.381614 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:36.382276 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:16:36.382319 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:16:36.384802 (+ 2483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:16:36.388208 (+ 3406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:36.388216 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:36.388221 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:36.388623 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:36.389058 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:36.389062 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:36.389066 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:36.389211 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:36.389831 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:16:36.389838 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:16:36.390044 (+ 206us) client_negotiation.cc:769] Sending connection context
0620 23:16:36.390223 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 23:16:36.390541 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":397,"threads_started":1}
I20260620 23:16:36.392513 1160 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.191.253" port: 46095 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:16:36 dist-test-slave-jkbq krb5kdc[932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
Jun 20 23:16:36 dist-test-slave-jkbq krb5kdc[932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997396, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.252@KRBTEST.COM for kudu/127.10.191.253@KRBTEST.COM
I20260620 23:16:36.406028 1165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.389721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33302 (local address 127.10.191.254:33619)
0620 23:16:36.390388 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:36.390392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:36.390410 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:16:36.390447 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:36.390451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:36.390515 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:36.390631 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:36.392207 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.393368 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.394990 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.395610 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.395696 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:36.400476 (+ 4780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:36.400481 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:36.400484 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:36.400514 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:36.403403 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:36.404041 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:36.404051 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:36.404053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:36.404175 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:36.404547 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:36.404556 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:36.404558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:36.404807 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:36.405076 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:36.405407 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 23:16:36.405621 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":342,"threads_started":1}
I20260620 23:16:36.406237 1164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.393066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:46095 (local address 127.0.0.1:55716)
0620 23:16:36.393342 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:36.393357 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:36.393450 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:36.393714 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:36.393718 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:36.394149 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:16:36.394326 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:36.394337 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.395528 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.395531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:36.396900 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:36.396917 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.397650 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.397657 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:36.397883 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:36.398408 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:16:36.398431 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:16:36.400769 (+ 2338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:16:36.403897 (+ 3128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:36.403903 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:36.403907 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:36.404272 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:36.404671 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:36.404676 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:36.404680 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:36.404823 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:36.405325 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:16:36.405330 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:16:36.405489 (+ 159us) client_negotiation.cc:769] Sending connection context
0620 23:16:36.405628 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 23:16:36.405894 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260620 23:16:36.406780 1090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.393244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55716 (local address 127.10.191.253:46095)
0620 23:16:36.393441 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:36.393450 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:36.393468 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:16:36.393539 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:36.393542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:36.393598 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:36.393668 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:36.394478 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.395380 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.397081 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.397490 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.397590 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:36.400978 (+ 3388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:36.400983 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:36.400986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:36.401018 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:36.403752 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:36.404405 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:36.404416 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:36.404418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:36.404558 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:36.404933 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:36.404938 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:36.404941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:36.405201 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:36.405365 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:36.405724 (+ 359us) server_negotiation.cc:299] Negotiation successful
0620 23:16:36.405934 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 23:16:36.407666 1160 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.191.252" port: 39993 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:16:36 dist-test-slave-jkbq krb5kdc[932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997396, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.252@KRBTEST.COM for kudu/127.10.191.252@KRBTEST.COM
Jun 20 23:16:36 dist-test-slave-jkbq krb5kdc[932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.253@KRBTEST.COM
I20260620 23:16:36.420504 1164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.408482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:39993 (local address 127.0.0.1:57228)
0620 23:16:36.408711 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:36.408723 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:36.408839 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:36.409123 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:36.409127 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:36.409483 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:16:36.409594 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:36.409601 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.410484 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.410491 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:36.411379 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:36.411389 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.412044 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.412050 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:36.412243 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:36.412802 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:16:36.412827 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:16:36.415017 (+ 2190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:16:36.418215 (+ 3198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:36.418221 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:36.418224 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:36.418562 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:36.418942 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:36.418947 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:36.418951 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:36.419075 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:36.419537 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:16:36.419541 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:16:36.419678 (+ 137us) client_negotiation.cc:769] Sending connection context
0620 23:16:36.419822 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 23:16:36.420111 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152}
I20260620 23:16:36.420651 1163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.408432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57228 (local address 127.10.191.252:39993)
0620 23:16:36.408601 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:36.408618 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:36.408793 (+ 175us) server_negotiation.cc:407] Connection header received
0620 23:16:36.408950 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:36.408954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:36.409013 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:36.409102 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:36.409720 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.410370 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.411506 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.411898 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.412031 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:36.415192 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:36.415197 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:36.415203 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:36.415237 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:36.418059 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:36.418674 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:36.418683 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:36.418686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:36.418807 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:36.419180 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:36.419187 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:36.419190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:36.419430 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:36.419572 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:36.419904 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 23:16:36.420259 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":140,"server-negotiator.queue_time_us":63}
I20260620 23:16:36.421892 1090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.407865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55722 (local address 127.10.191.253:46095)
0620 23:16:36.408014 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:36.408019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:36.408079 (+ 60us) server_negotiation.cc:407] Connection header received
0620 23:16:36.408272 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:36.408277 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:36.408350 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:36.408469 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:36.409579 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.410277 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.411580 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.412001 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.412133 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:36.416652 (+ 4519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:36.416658 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:36.416662 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:36.416698 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:36.419163 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:36.419786 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:36.419792 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:36.419797 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:36.419928 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:36.420297 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:36.420302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:36.420306 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:36.420553 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:36.420855 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:36.421328 (+ 473us) server_negotiation.cc:299] Negotiation successful
0620 23:16:36.421540 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 23:16:36.426681 1160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73fe2761fc1a4204a26373cbc45abd1e: Bootstrap starting.
I20260620 23:16:36.426746 11007 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:16:36.430481 1160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73fe2761fc1a4204a26373cbc45abd1e: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:36.431586 1160 log.cc:824] T 00000000000000000000000000000000 P 73fe2761fc1a4204a26373cbc45abd1e: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:36.430114 987 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b768f9fc187c4c33814a4f02b55689e6"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33302
I20260620 23:16:36.434181 1160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73fe2761fc1a4204a26373cbc45abd1e: No bootstrap required, opened a new log
I20260620 23:16:36.441151 1160 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73fe2761fc1a4204a26373cbc45abd1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b768f9fc187c4c33814a4f02b55689e6" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 33619 } } peers { permanent_uuid: "41129b9f87b94019b56b19ad796771f1" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 46095 } } peers { permanent_uuid: "73fe2761fc1a4204a26373cbc45abd1e" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 39993 } }
I20260620 23:16:36.441478 1160 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73fe2761fc1a4204a26373cbc45abd1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73fe2761fc1a4204a26373cbc45abd1e, State: Initialized, Role: FOLLOWER
I20260620 23:16:36.441921 1160 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73fe2761fc1a4204a26373cbc45abd1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b768f9fc187c4c33814a4f02b55689e6" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 33619 } } peers { permanent_uuid: "41129b9f87b94019b56b19ad796771f1" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 46095 } } peers { permanent_uuid: "73fe2761fc1a4204a26373cbc45abd1e" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 39993 } }
I20260620 23:16:36.442883 1167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73fe2761fc1a4204a26373cbc45abd1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b768f9fc187c4c33814a4f02b55689e6" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 33619 } } peers { permanent_uuid: "41129b9f87b94019b56b19ad796771f1" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 46095 } } peers { permanent_uuid: "73fe2761fc1a4204a26373cbc45abd1e" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 39993 } } }
I20260620 23:16:36.443254 1167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73fe2761fc1a4204a26373cbc45abd1e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:16:36.444027 1160 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73fe2761fc1a4204a26373cbc45abd1e [sys.catalog]: configured and running, proceeding with master startup.
W20260620 23:16:36.453473 1178 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 73fe2761fc1a4204a26373cbc45abd1e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:16:36.453601 1178 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:16:36.451128 1093 master_runner.cc:186] Error getting master registration for 127.10.191.254:33619: OK, instance_id { permanent_uuid: "b768f9fc187c4c33814a4f02b55689e6" instance_seqno: 1781997395370674 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 23:16:36.458448 1093 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 23:16:36.510356 1012 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.191.252" port: 39993 } attempt: 4
Jun 20 23:16:36 dist-test-slave-jkbq krb5kdc[932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997395, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for kudu/127.10.191.252@KRBTEST.COM
I20260620 23:16:36.527241 1016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.511166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:39993 (local address 127.0.0.1:57234)
0620 23:16:36.511355 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:36.511378 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:36.511486 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:36.511850 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:36.511854 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:36.512439 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:16:36.512673 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:36.512687 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.514129 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.514134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:36.515632 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:36.515649 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.516705 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.516710 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:36.516939 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:36.517629 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:16:36.517651 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:16:36.520289 (+ 2638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:16:36.524528 (+ 4239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:36.524533 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:36.524538 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:36.525039 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:36.525434 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:36.525439 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:36.525443 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:36.525600 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:36.526240 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:16:36.526245 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:16:36.526409 (+ 164us) client_negotiation.cc:769] Sending connection context
0620 23:16:36.526555 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 23:16:36.526826 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 23:16:36.527279 1163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.511239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57234 (local address 127.10.191.252:39993)
0620 23:16:36.511481 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:36.511487 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:36.511511 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:16:36.511582 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:36.511587 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:36.511667 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:36.511779 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:36.512855 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.513984 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.515854 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.516563 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.516692 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:36.520473 (+ 3781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:36.520481 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:36.520486 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:36.520526 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:36.524398 (+ 3872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:36.525162 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:36.525168 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:36.525172 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:36.525319 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:36.525709 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:36.525713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:36.525716 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:36.526102 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:36.526262 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:36.526667 (+ 405us) server_negotiation.cc:299] Negotiation successful
0620 23:16:36.526909 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 23:16:36.533335 1012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b768f9fc187c4c33814a4f02b55689e6: Bootstrap starting.
I20260620 23:16:36.537326 1012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b768f9fc187c4c33814a4f02b55689e6: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:36.538635 1012 log.cc:824] T 00000000000000000000000000000000 P b768f9fc187c4c33814a4f02b55689e6: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:36.541929 1012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b768f9fc187c4c33814a4f02b55689e6: No bootstrap required, opened a new log
I20260620 23:16:36.549526 1012 raft_consensus.cc:348] T 00000000000000000000000000000000 P b768f9fc187c4c33814a4f02b55689e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b768f9fc187c4c33814a4f02b55689e6" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 33619 } } peers { permanent_uuid: "41129b9f87b94019b56b19ad796771f1" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 46095 } } peers { permanent_uuid: "73fe2761fc1a4204a26373cbc45abd1e" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 39993 } }
I20260620 23:16:36.549930 1012 raft_consensus.cc:729] T 00000000000000000000000000000000 P b768f9fc187c4c33814a4f02b55689e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b768f9fc187c4c33814a4f02b55689e6, State: Initialized, Role: FOLLOWER
I20260620 23:16:36.550431 1012 consensus_queue.cc:260] T 00000000000000000000000000000000 P b768f9fc187c4c33814a4f02b55689e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b768f9fc187c4c33814a4f02b55689e6" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 33619 } } peers { permanent_uuid: "41129b9f87b94019b56b19ad796771f1" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 46095 } } peers { permanent_uuid: "73fe2761fc1a4204a26373cbc45abd1e" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 39993 } }
I20260620 23:16:36.551443 1181 sys_catalog.cc:455] T 00000000000000000000000000000000 P b768f9fc187c4c33814a4f02b55689e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b768f9fc187c4c33814a4f02b55689e6" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 33619 } } peers { permanent_uuid: "41129b9f87b94019b56b19ad796771f1" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 46095 } } peers { permanent_uuid: "73fe2761fc1a4204a26373cbc45abd1e" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 39993 } } }
I20260620 23:16:36.551831 1181 sys_catalog.cc:458] T 00000000000000000000000000000000 P b768f9fc187c4c33814a4f02b55689e6 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:16:36.552721 1012 sys_catalog.cc:564] T 00000000000000000000000000000000 P b768f9fc187c4c33814a4f02b55689e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:16:36.552780 987 raft_consensus.cc:482] T 00000000000000000000000000000000 P b768f9fc187c4c33814a4f02b55689e6 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 23:16:36.552994 987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b768f9fc187c4c33814a4f02b55689e6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:36.555570 987 raft_consensus.cc:504] T 00000000000000000000000000000000 P b768f9fc187c4c33814a4f02b55689e6 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b768f9fc187c4c33814a4f02b55689e6" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 33619 } } peers { permanent_uuid: "41129b9f87b94019b56b19ad796771f1" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 46095 } } peers { permanent_uuid: "73fe2761fc1a4204a26373cbc45abd1e" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 39993 } }
I20260620 23:16:36.556880 987 leader_election.cc:288] T 00000000000000000000000000000000 P b768f9fc187c4c33814a4f02b55689e6 [CANDIDATE]: Term 1 election: Requested vote from peers 41129b9f87b94019b56b19ad796771f1 (127.10.191.253:46095), 73fe2761fc1a4204a26373cbc45abd1e (127.10.191.252:39993)
I20260620 23:16:36.557828 1135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b768f9fc187c4c33814a4f02b55689e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "73fe2761fc1a4204a26373cbc45abd1e"
I20260620 23:16:36.558318 1135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73fe2761fc1a4204a26373cbc45abd1e [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:36.561846 1135 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 73fe2761fc1a4204a26373cbc45abd1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b768f9fc187c4c33814a4f02b55689e6 in term 1.
I20260620 23:16:36.562680 963 leader_election.cc:302] T 00000000000000000000000000000000 P b768f9fc187c4c33814a4f02b55689e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73fe2761fc1a4204a26373cbc45abd1e, b768f9fc187c4c33814a4f02b55689e6; no voters:
I20260620 23:16:36.563323 1181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b768f9fc187c4c33814a4f02b55689e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:16:36.576800 1181 raft_consensus.cc:686] T 00000000000000000000000000000000 P b768f9fc187c4c33814a4f02b55689e6 [term 1 LEADER]: Becoming Leader. State: Replica: b768f9fc187c4c33814a4f02b55689e6, State: Running, Role: LEADER
I20260620 23:16:36.578388 1181 consensus_queue.cc:237] T 00000000000000000000000000000000 P b768f9fc187c4c33814a4f02b55689e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b768f9fc187c4c33814a4f02b55689e6" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 33619 } } peers { permanent_uuid: "41129b9f87b94019b56b19ad796771f1" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 46095 } } peers { permanent_uuid: "73fe2761fc1a4204a26373cbc45abd1e" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 39993 } }
I20260620 23:16:36.585100 1086 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.191.252" port: 39993 } attempt: 5
W20260620 23:16:36.593262 1114 tablet.cc:2369] T 00000000000000000000000000000000 P 73fe2761fc1a4204a26373cbc45abd1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:16:36.594797 1187 sys_catalog.cc:455] T 00000000000000000000000000000000 P b768f9fc187c4c33814a4f02b55689e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b768f9fc187c4c33814a4f02b55689e6. Latest consensus state: current_term: 1 leader_uuid: "b768f9fc187c4c33814a4f02b55689e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b768f9fc187c4c33814a4f02b55689e6" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 33619 } } peers { permanent_uuid: "41129b9f87b94019b56b19ad796771f1" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 46095 } } peers { permanent_uuid: "73fe2761fc1a4204a26373cbc45abd1e" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 39993 } } }
I20260620 23:16:36.595328 1187 sys_catalog.cc:458] T 00000000000000000000000000000000 P b768f9fc187c4c33814a4f02b55689e6 [sys.catalog]: This master's current role is: LEADER
Jun 20 23:16:36 dist-test-slave-jkbq krb5kdc[932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997395, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.253@KRBTEST.COM for kudu/127.10.191.252@KRBTEST.COM
W20260620 23:16:36.610248 966 tablet.cc:2369] T 00000000000000000000000000000000 P b768f9fc187c4c33814a4f02b55689e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:16:36.611827 1089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.585865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:39993 (local address 127.0.0.1:57240)
0620 23:16:36.591469 (+ 5604us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:36.591490 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:36.591655 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:36.592363 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:36.592371 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:36.593054 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:16:36.593272 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:36.593286 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.594995 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.595003 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:36.596920 (+ 1917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:36.596940 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.598613 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.598622 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:36.599023 (+ 401us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:36.599837 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:16:36.599869 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:16:36.603076 (+ 3207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:16:36.608146 (+ 5070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:36.608155 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:36.608161 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:36.608818 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:36.609337 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:36.609345 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:36.609352 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:36.609590 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:36.610427 (+ 837us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:16:36.610435 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:16:36.610643 (+ 208us) client_negotiation.cc:769] Sending connection context
0620 23:16:36.610845 (+ 202us) client_negotiation.cc:240] Negotiation successful
0620 23:16:36.611260 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5491}
I20260620 23:16:36.613098 1163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.590537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57240 (local address 127.10.191.252:39993)
0620 23:16:36.591058 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:36.591067 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:36.591796 (+ 729us) server_negotiation.cc:407] Connection header received
0620 23:16:36.591894 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:36.591902 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:36.592018 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:36.592127 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:36.593470 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.594747 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.597142 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.598143 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.598291 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:36.603293 (+ 5002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:36.603302 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:36.603309 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:36.603361 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:36.607947 (+ 4586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:36.608992 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:36.609001 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:36.609007 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:36.609186 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:36.609738 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:36.609746 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:36.609752 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:36.610184 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:36.610350 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:36.612278 (+ 1928us) server_negotiation.cc:299] Negotiation successful
0620 23:16:36.612583 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356}
I20260620 23:16:36.618957 1086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41129b9f87b94019b56b19ad796771f1: Bootstrap starting.
I20260620 23:16:36.623119 1086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41129b9f87b94019b56b19ad796771f1: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:36.622921 1198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:16:36.624404 1086 log.cc:824] T 00000000000000000000000000000000 P 41129b9f87b94019b56b19ad796771f1: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:36.636531 1086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41129b9f87b94019b56b19ad796771f1: No bootstrap required, opened a new log
I20260620 23:16:36.651465 1086 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41129b9f87b94019b56b19ad796771f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b768f9fc187c4c33814a4f02b55689e6" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 33619 } } peers { permanent_uuid: "41129b9f87b94019b56b19ad796771f1" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 46095 } } peers { permanent_uuid: "73fe2761fc1a4204a26373cbc45abd1e" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 39993 } }
I20260620 23:16:36.651858 1086 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41129b9f87b94019b56b19ad796771f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41129b9f87b94019b56b19ad796771f1, State: Initialized, Role: FOLLOWER
I20260620 23:16:36.652423 1086 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41129b9f87b94019b56b19ad796771f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b768f9fc187c4c33814a4f02b55689e6" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 33619 } } peers { permanent_uuid: "41129b9f87b94019b56b19ad796771f1" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 46095 } } peers { permanent_uuid: "73fe2761fc1a4204a26373cbc45abd1e" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 39993 } }
I20260620 23:16:36.659364 1205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41129b9f87b94019b56b19ad796771f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b768f9fc187c4c33814a4f02b55689e6" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 33619 } } peers { permanent_uuid: "41129b9f87b94019b56b19ad796771f1" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 46095 } } peers { permanent_uuid: "73fe2761fc1a4204a26373cbc45abd1e" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 39993 } } }
I20260620 23:16:36.659751 1205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41129b9f87b94019b56b19ad796771f1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:16:36.661450 1086 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41129b9f87b94019b56b19ad796771f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:16:36.661605 1061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b768f9fc187c4c33814a4f02b55689e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "41129b9f87b94019b56b19ad796771f1"
I20260620 23:16:36.662140 1198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:16:36.662648 1061 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41129b9f87b94019b56b19ad796771f1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:36.666301 1061 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 41129b9f87b94019b56b19ad796771f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b768f9fc187c4c33814a4f02b55689e6 in term 1.
I20260620 23:16:36.675683 1165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.591282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33312 (local address 127.10.191.254:33619)
0620 23:16:36.591472 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:36.591479 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:36.608034 (+ 16555us) server_negotiation.cc:407] Connection header received
0620 23:16:36.608499 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:36.608508 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:36.608610 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:36.608700 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:36.631287 (+ 22587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.632704 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.646944 (+ 14240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.647696 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.647836 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:36.668535 (+ 20699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:36.668545 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:36.668553 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:36.668610 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:36.672411 (+ 3801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:36.673279 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:36.673286 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:36.673289 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:36.673460 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:36.673891 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:36.673896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:36.673899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:36.674265 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:36.674503 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:36.675084 (+ 581us) server_negotiation.cc:299] Negotiation successful
0620 23:16:36.675331 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 23:16:36.676314 945 master_runner.cc:186] Error getting master registration for 127.10.191.253:46095: OK, instance_id { permanent_uuid: "41129b9f87b94019b56b19ad796771f1" instance_seqno: 1781997395860732 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 23:16:36.684866 945 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 23:16:36.685299 1196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.590715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57244 (local address 127.10.191.252:39993)
0620 23:16:36.592334 (+ 1619us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:36.592339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:36.597054 (+ 4715us) server_negotiation.cc:407] Connection header received
0620 23:16:36.597590 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:36.597598 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:36.597689 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:36.597774 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:36.619375 (+ 21601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.621117 (+ 1742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.638669 (+ 17552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.639659 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.639935 (+ 276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:36.649308 (+ 9373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:36.649316 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:36.649323 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:36.649368 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:36.653447 (+ 4079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:36.666230 (+ 12783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:36.666237 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:36.666241 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:36.666442 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:36.673317 (+ 6875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:36.673323 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:36.673326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:36.682418 (+ 9092us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:36.682684 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:36.684496 (+ 1812us) server_negotiation.cc:299] Negotiation successful
0620 23:16:36.684824 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1480,"thread_start_us":7691,"threads_started":1}
I20260620 23:16:36.698526 1090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:36.607476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55728 (local address 127.10.191.253:46095)
0620 23:16:36.607667 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:36.607676 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:36.647243 (+ 39567us) server_negotiation.cc:407] Connection header received
0620 23:16:36.650075 (+ 2832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:36.650083 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:36.650446 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:36.650522 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:36.654385 (+ 3863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.655567 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.686682 (+ 31115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:36.687405 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:36.687540 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:36.691123 (+ 3583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:36.691132 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:36.691137 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:36.691191 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:36.694802 (+ 3611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:36.695654 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:36.695661 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:36.695664 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:36.695831 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:36.696713 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:36.696719 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:36.696722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:36.697088 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:36.697318 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:36.697905 (+ 587us) server_negotiation.cc:299] Negotiation successful
0620 23:16:36.698159 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 23:16:36.706748 1135 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 73fe2761fc1a4204a26373cbc45abd1e [term 1 FOLLOWER]: Refusing update from remote peer b768f9fc187c4c33814a4f02b55689e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:16:36.708464 1061 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 41129b9f87b94019b56b19ad796771f1 [term 1 FOLLOWER]: Refusing update from remote peer b768f9fc187c4c33814a4f02b55689e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:16:36.709479 1181 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b768f9fc187c4c33814a4f02b55689e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73fe2761fc1a4204a26373cbc45abd1e" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 39993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:16:36.710557 1181 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b768f9fc187c4c33814a4f02b55689e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41129b9f87b94019b56b19ad796771f1" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 46095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:16:36.732018 1205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41129b9f87b94019b56b19ad796771f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b768f9fc187c4c33814a4f02b55689e6. Latest consensus state: current_term: 1 leader_uuid: "b768f9fc187c4c33814a4f02b55689e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b768f9fc187c4c33814a4f02b55689e6" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 33619 } } peers { permanent_uuid: "41129b9f87b94019b56b19ad796771f1" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 46095 } } peers { permanent_uuid: "73fe2761fc1a4204a26373cbc45abd1e" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 39993 } } }
I20260620 23:16:36.732364 1205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41129b9f87b94019b56b19ad796771f1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:16:36.733737 1167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73fe2761fc1a4204a26373cbc45abd1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b768f9fc187c4c33814a4f02b55689e6. Latest consensus state: current_term: 1 leader_uuid: "b768f9fc187c4c33814a4f02b55689e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b768f9fc187c4c33814a4f02b55689e6" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 33619 } } peers { permanent_uuid: "41129b9f87b94019b56b19ad796771f1" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 46095 } } peers { permanent_uuid: "73fe2761fc1a4204a26373cbc45abd1e" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 39993 } } }
I20260620 23:16:36.734140 1167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73fe2761fc1a4204a26373cbc45abd1e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:16:36.740665 1218 mvcc.cc:205] Tried to move back new op lower bound from 7299061336890871808 to 7299061336400977920. Current Snapshot: MvccSnapshot[applied={T|T < 7299061336890871808}]
I20260620 23:16:36.748924 1205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41129b9f87b94019b56b19ad796771f1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b768f9fc187c4c33814a4f02b55689e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b768f9fc187c4c33814a4f02b55689e6" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 33619 } } peers { permanent_uuid: "41129b9f87b94019b56b19ad796771f1" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 46095 } } peers { permanent_uuid: "73fe2761fc1a4204a26373cbc45abd1e" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 39993 } } }
I20260620 23:16:36.749169 1205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41129b9f87b94019b56b19ad796771f1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:16:36.753065 1222 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 23:16:36.753273 1167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73fe2761fc1a4204a26373cbc45abd1e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b768f9fc187c4c33814a4f02b55689e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b768f9fc187c4c33814a4f02b55689e6" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 33619 } } peers { permanent_uuid: "41129b9f87b94019b56b19ad796771f1" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 46095 } } peers { permanent_uuid: "73fe2761fc1a4204a26373cbc45abd1e" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 39993 } } }
I20260620 23:16:36.753539 1167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73fe2761fc1a4204a26373cbc45abd1e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:16:36.755923 1187 sys_catalog.cc:455] T 00000000000000000000000000000000 P b768f9fc187c4c33814a4f02b55689e6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b768f9fc187c4c33814a4f02b55689e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b768f9fc187c4c33814a4f02b55689e6" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 33619 } } peers { permanent_uuid: "41129b9f87b94019b56b19ad796771f1" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 46095 } } peers { permanent_uuid: "73fe2761fc1a4204a26373cbc45abd1e" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 39993 } } }
I20260620 23:16:36.756214 1187 sys_catalog.cc:458] T 00000000000000000000000000000000 P b768f9fc187c4c33814a4f02b55689e6 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:36.756314 1198 catalog_manager.cc:1347] Generated new cluster ID: 62a1795baaa1454792cd1fdc5be31bcf
I20260620 23:16:36.756422 1198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:16:36.756549 1187 sys_catalog.cc:455] T 00000000000000000000000000000000 P b768f9fc187c4c33814a4f02b55689e6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b768f9fc187c4c33814a4f02b55689e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b768f9fc187c4c33814a4f02b55689e6" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 33619 } } peers { permanent_uuid: "41129b9f87b94019b56b19ad796771f1" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 46095 } } peers { permanent_uuid: "73fe2761fc1a4204a26373cbc45abd1e" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 39993 } } }
I20260620 23:16:36.756755 1187 sys_catalog.cc:458] T 00000000000000000000000000000000 P b768f9fc187c4c33814a4f02b55689e6 [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:36.781142 1198 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:16:36.784404 1198 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:16:36.814177 1198 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b768f9fc187c4c33814a4f02b55689e6: Generated new TSK 0
I20260620 23:16:36.815577 1198 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 23:16:36.923014 1165 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:16:36.913332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33318 (local address 127.10.191.254:33619)
0620 23:16:36.913540 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:36.913547 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:36.920661 (+ 7114us) server_negotiation.cc:407] Connection header received
0620 23:16:36.920974 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:36.920982 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:36.921090 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:36.921190 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:36.922616 (+ 1426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33318: BlockingRecv error: recv got EOF from 127.0.0.1:33318 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 23:16:36.923688 1196 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:16:36.913722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57246 (local address 127.10.191.252:39993)
0620 23:16:36.913935 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:36.913941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:36.920475 (+ 6534us) server_negotiation.cc:407] Connection header received
0620 23:16:36.921828 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:36.921834 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:36.921941 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:36.922018 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:36.923434 (+ 1416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57246: BlockingRecv error: recv got EOF from 127.0.0.1:57246 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260620 23:16:36.925562 1090 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:16:36.920990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55742 (local address 127.10.191.253:46095)
0620 23:16:36.921213 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:36.921219 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:36.923654 (+ 2435us) server_negotiation.cc:407] Connection header received
0620 23:16:36.923928 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:36.923936 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:36.924058 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:36.924183 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:36.925259 (+ 1076us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55742: BlockingRecv error: recv got EOF from 127.0.0.1:55742 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260620 23:16:36.933128 1090 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:16:36.930938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55758 (local address 127.10.191.253:46095)
0620 23:16:36.931139 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:36.931146 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:36.931807 (+ 661us) server_negotiation.cc:407] Connection header received
0620 23:16:36.932035 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:36.932040 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:36.932135 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:36.932243 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:36.932921 (+ 678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55758: BlockingRecv error: recv got EOF from 127.0.0.1:55758 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 23:16:36.944900 1090 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:16:36.937920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55766 (local address 127.10.191.253:46095)
0620 23:16:36.938139 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:36.938147 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:36.943370 (+ 5223us) server_negotiation.cc:407] Connection header received
0620 23:16:36.943575 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:36.943579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:36.943697 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:36.943811 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:36.944693 (+ 882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55766: BlockingRecv error: recv got EOF from 127.0.0.1:55766 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260620 23:16:36.951681 1090 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:16:36.949635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55782 (local address 127.10.191.253:46095)
0620 23:16:36.949829 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:36.949837 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:36.950318 (+ 481us) server_negotiation.cc:407] Connection header received
0620 23:16:36.950508 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:36.950513 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:36.950600 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:36.950711 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:36.951454 (+ 743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55782: BlockingRecv error: recv got EOF from 127.0.0.1:55782 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260620 23:16:36.960443 1090 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:16:36.958457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55796 (local address 127.10.191.253:46095)
0620 23:16:36.958692 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:36.958701 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:36.959220 (+ 519us) server_negotiation.cc:407] Connection header received
0620 23:16:36.959412 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:36.959424 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:36.959498 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:36.959602 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:36.960235 (+ 633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55796: BlockingRecv error: recv got EOF from 127.0.0.1:55796 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
W20260620 23:16:36.973264 1090 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:16:36.971254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55800 (local address 127.10.191.253:46095)
0620 23:16:36.971469 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:36.971478 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:36.971957 (+ 479us) server_negotiation.cc:407] Connection header received
0620 23:16:36.972161 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:36.972167 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:36.972242 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:36.972347 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:36.973028 (+ 681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55800: BlockingRecv error: recv got EOF from 127.0.0.1:55800 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 23:16:36.994532 1090 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:16:36.992341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55808 (local address 127.10.191.253:46095)
0620 23:16:36.992574 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:36.992584 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:36.993160 (+ 576us) server_negotiation.cc:407] Connection header received
0620 23:16:36.993379 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:36.993383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:36.993456 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:36.993564 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:36.994300 (+ 736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55808: BlockingRecv error: recv got EOF from 127.0.0.1:55808 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 23:16:37.032419 1090 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:16:37.030093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55816 (local address 127.10.191.253:46095)
0620 23:16:37.030345 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:37.030355 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:37.030941 (+ 586us) server_negotiation.cc:407] Connection header received
0620 23:16:37.031194 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:37.031205 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:37.031288 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:37.031405 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:37.032156 (+ 751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55816: BlockingRecv error: recv got EOF from 127.0.0.1:55816 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260620 23:16:37.102336 1090 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:16:37.100057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55822 (local address 127.10.191.253:46095)
0620 23:16:37.100316 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:37.100326 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:37.100930 (+ 604us) server_negotiation.cc:407] Connection header received
0620 23:16:37.101079 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:37.101085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:37.101196 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:37.101351 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:37.102069 (+ 718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55822: BlockingRecv error: recv got EOF from 127.0.0.1:55822 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260620 23:16:37.236157 1090 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:16:37.234043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55830 (local address 127.10.191.253:46095)
0620 23:16:37.234278 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:37.234285 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:37.234625 (+ 340us) server_negotiation.cc:407] Connection header received
0620 23:16:37.234880 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:37.234885 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:37.234959 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:37.235071 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:37.235812 (+ 741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55830: BlockingRecv error: recv got EOF from 127.0.0.1:55830 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 23:16:37.455714 1178 catalog_manager.cc:1259] Loaded cluster ID: 62a1795baaa1454792cd1fdc5be31bcf
I20260620 23:16:37.455850 1178 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 73fe2761fc1a4204a26373cbc45abd1e: loading cluster ID for follower catalog manager: success
I20260620 23:16:37.458359 1178 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 73fe2761fc1a4204a26373cbc45abd1e: acquiring CA information for follower catalog manager: success
I20260620 23:16:37.460117 1178 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 73fe2761fc1a4204a26373cbc45abd1e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 23:16:37.498445 1090 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:16:37.496662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55844 (local address 127.10.191.253:46095)
0620 23:16:37.496923 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:37.496930 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:37.497110 (+ 180us) server_negotiation.cc:407] Connection header received
0620 23:16:37.497343 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:37.497347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:37.497419 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:37.497532 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:37.498278 (+ 746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55844: BlockingRecv error: recv got EOF from 127.0.0.1:55844 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 23:16:37.759694 1222 catalog_manager.cc:1259] Loaded cluster ID: 62a1795baaa1454792cd1fdc5be31bcf
I20260620 23:16:37.759826 1222 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 41129b9f87b94019b56b19ad796771f1: loading cluster ID for follower catalog manager: success
I20260620 23:16:37.763232 1222 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 41129b9f87b94019b56b19ad796771f1: acquiring CA information for follower catalog manager: success
I20260620 23:16:37.765095 1222 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 41129b9f87b94019b56b19ad796771f1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:16:38.021409 1293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:38.014787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55848 (local address 127.10.191.253:46095)
0620 23:16:38.015633 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:38.015641 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:38.015657 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:16:38.015763 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:38.015768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:38.015967 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:38.016124 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:16:38.016589 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:38.017558 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:38.018959 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:38.019416 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:38.019571 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:38.020084 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:16:38.020696 (+ 612us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:16:38.020862 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:38.020973 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 23:16:38.021061 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":705,"thread_start_us":528,"threads_started":1}
I20260620 23:16:38.034216 1299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:38.028167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57250 (local address 127.10.191.252:39993)
0620 23:16:38.029064 (+ 897us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:38.029069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:38.029084 (+ 15us) server_negotiation.cc:407] Connection header received
0620 23:16:38.029228 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:38.029231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:38.029392 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:38.029565 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:16:38.029961 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:38.030675 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:38.032044 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:38.032452 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:38.032590 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:38.033092 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:16:38.033602 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:16:38.033774 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:38.033862 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 23:16:38.033944 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":802,"thread_start_us":711,"threads_started":1}
I20260620 23:16:38.059264 1299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:38.045139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57264 (local address 127.10.191.252:39993)
0620 23:16:38.045368 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:38.045376 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:38.051578 (+ 6202us) server_negotiation.cc:407] Connection header received
0620 23:16:38.051914 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:38.051919 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:38.052007 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:38.052074 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:16:38.053079 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:38.054282 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:38.056480 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:38.057115 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:38.057244 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:38.058047 (+ 803us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:16:38.058183 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:16:38.058266 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:38.058847 (+ 581us) server_negotiation.cc:299] Negotiation successful
0620 23:16:38.058955 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 23:16:38.061633 1293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:38.052336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55858 (local address 127.10.191.253:46095)
0620 23:16:38.052573 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:38.052582 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:38.053659 (+ 1077us) server_negotiation.cc:407] Connection header received
0620 23:16:38.053973 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:38.053982 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:38.054107 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:38.054200 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:16:38.055330 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:38.056787 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:38.059038 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:38.059695 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:38.059812 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:38.060815 (+ 1003us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:16:38.060941 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:16:38.061009 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:38.061239 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 23:16:38.061321 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 23:16:38.064309 1304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:38.044986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33322 (local address 127.10.191.254:33619)
0620 23:16:38.057021 (+ 12035us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:38.057027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:38.057045 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:16:38.057165 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:38.057171 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:38.057395 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:38.057582 (+ 187us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:16:38.058382 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:38.059710 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:38.061975 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:38.062594 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:38.062724 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:38.063403 (+ 679us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:16:38.063730 (+ 327us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:16:38.063868 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:38.063969 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 23:16:38.064055 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11904,"thread_start_us":12378,"threads_started":1}
I20260620 23:16:38.069288 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 945
I20260620 23:16:38.070886 1007 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:16:38.183470 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 945
W20260620 23:16:38.203999 1111 connection.cc:535] server connection from 127.0.0.1:57234 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 23:16:38.204413 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 1019
I20260620 23:16:38.205577 1081 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:16:38.313743 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 1019
I20260620 23:16:38.333403 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 1093
I20260620 23:16:38.334429 1155 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:16:38.440016 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 1093
2026-06-20T23:16:38Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4846 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:16:38 dist-test-slave-jkbq krb5kdc[1313](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:16:38 dist-test-slave-jkbq krb5kdc[1313](info): set up 2 sockets
Jun 20 23:16:38 dist-test-slave-jkbq krb5kdc[1313](info): commencing operation
krb5kdc: starting...
W20260620 23:16:39.551453 11007 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.008s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/krb5kdc/test-user.keytab.
Jun 20 23:16:39 dist-test-slave-jkbq krb5kdc[1313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:16:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:16:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.191.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
Entry for principal kudu/127.10.191.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab.
I20260620 23:16:39.794363 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:39843
--webserver_interface=127.10.191.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:46105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:39843,127.10.191.253:42935,127.10.191.252:36993
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:16:40.116307 1326 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:16:40.116634 1326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:40.116773 1326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:40.116896 1326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:40.124369 1326 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:16:40.124460 1326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:40.124547 1326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:40.124622 1326 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:16:40.124711 1326 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:16:40.134521 1326 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:46105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:39843,127.10.191.253:42935,127.10.191.252:36993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.254.keytab
--principal=kudu/127.10.191.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.254:39843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.191.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:16:40.135823 1326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:40.137121 1326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:40.149257 1332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:40.149317 1331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:40.149341 1334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:40.150437 1326 server_base.cc:1034] running on GCE node
I20260620 23:16:40.151165 1326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:40.152428 1326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:40.153642 1326 hybrid_clock.cc:648] HybridClock initialized: now 1781997400153581 us; error 31 us; skew 500 ppm
I20260620 23:16:40.153970 1326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:16:40 dist-test-slave-jkbq krb5kdc[1313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997400, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:40.157074 1326 init.cc:377] Logged in from keytab as kudu/127.10.191.254@KRBTEST.COM (short username kudu)
I20260620 23:16:40.158519 1326 webserver.cc:460] Webserver started at http://127.10.191.254:43853/ using document root <none> and password file <none>
I20260620 23:16:40.159332 1326 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:40.159461 1326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:40.159808 1326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:40.162663 1326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-0/data/instance:
uuid: "158c6280f017418b9911d4359f39fa1b"
format_stamp: "Formatted at 2026-06-20 23:16:40 on dist-test-slave-jkbq"
server_key: "d31cbadf84d7b67ba39a558a45c227e8"
server_key_iv: "2d6769b4dd17b8e8858c74ef6198d74f"
server_key_version: "encryptionkey@0"
I20260620 23:16:40.163508 1326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-0/wal/instance:
uuid: "158c6280f017418b9911d4359f39fa1b"
format_stamp: "Formatted at 2026-06-20 23:16:40 on dist-test-slave-jkbq"
server_key: "d31cbadf84d7b67ba39a558a45c227e8"
server_key_iv: "2d6769b4dd17b8e8858c74ef6198d74f"
server_key_version: "encryptionkey@0"
I20260620 23:16:40.168412 1326 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 23:16:40.171713 1341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:40.172922 1326 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:16:40.173103 1326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-0/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-0/wal
uuid: "158c6280f017418b9911d4359f39fa1b"
format_stamp: "Formatted at 2026-06-20 23:16:40 on dist-test-slave-jkbq"
server_key: "d31cbadf84d7b67ba39a558a45c227e8"
server_key_iv: "2d6769b4dd17b8e8858c74ef6198d74f"
server_key_version: "encryptionkey@0"
I20260620 23:16:40.173324 1326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:40.192509 1326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:40.197105 1326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:40.197386 1326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:40.215488 1326 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.254:39843
I20260620 23:16:40.215508 1392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.254:39843 every 8 connection(s)
I20260620 23:16:40.216847 1326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-0/data/info.pb
I20260620 23:16:40.217768 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 1326
I20260620 23:16:40.217967 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-0/wal/instance
I20260620 23:16:40.218364 11007 external_mini_cluster.cc:1428] Setting key f93690f5aefd9c5189b07fa06fe80dc2
I20260620 23:16:40.220598 1393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:40.227103 1393 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39843 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.10.191.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.253@KRBTEST.COM" created.
Jun 20 23:16:40 dist-test-slave-jkbq krb5kdc[1313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997400, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:16:40.250842 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.230092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39843 (local address 127.0.0.1:43602)
0620 23:16:40.230934 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:40.230977 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:40.232898 (+ 1921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:40.233717 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:40.233724 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:40.234431 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:16:40.234767 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:40.234784 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.236341 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.236348 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:40.237397 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:40.237409 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.238291 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.238298 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:40.238526 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:40.239241 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:16:40.239263 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:16:40.243584 (+ 4321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:16:40.246737 (+ 3153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:40.246749 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:40.246760 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:40.247218 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:40.247599 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:40.247606 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:40.247611 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:40.247771 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:40.248498 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:16:40.248505 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:16:40.248742 (+ 237us) client_negotiation.cc:769] Sending connection context
0620 23:16:40.248981 (+ 239us) client_negotiation.cc:240] Negotiation successful
0620 23:16:40.249536 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":135,"thread_start_us":344,"threads_started":1}
I20260620 23:16:40.250877 1397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.230092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43602 (local address 127.10.191.254:39843)
0620 23:16:40.231716 (+ 1624us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:40.231724 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:40.231772 (+ 48us) server_negotiation.cc:407] Connection header received
0620 23:16:40.233265 (+ 1493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:40.233271 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:40.233495 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:40.233692 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:40.235000 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.236180 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.237535 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.238058 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.238257 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:40.243803 (+ 5546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:40.243811 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:40.243822 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:40.243851 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:40.246587 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:40.247346 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:40.247358 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:40.247360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:40.247494 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:40.247882 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:40.247891 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:40.247894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:40.248365 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:40.248569 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:40.249111 (+ 542us) server_negotiation.cc:299] Negotiation successful
0620 23:16:40.249420 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":1415,"thread_start_us":1216,"threads_started":1}
I20260620 23:16:40.253961 1393 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.191.253" port: 42935 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:16:40.255173 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.254741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:42935 (local address 127.0.0.1:56362)
0620 23:16:40.254965 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:40.255064 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.253:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 23:16:40.255451 1344 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.191.253:42935: connect: Connection refused (error 111)
I20260620 23:16:40.256920 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.256599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:42935 (local address 127.0.0.1:56376)
0620 23:16:40.256749 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:40.256811 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.253:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 23:16:40.257129 1393 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.191.253:42935: Network error: Client connection negotiation failed: client connection to 127.10.191.253:42935: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.253.keytab.
Entry for principal kudu/127.10.191.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.253.keytab.
I20260620 23:16:40.271898 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.253:42935
--webserver_interface=127.10.191.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.253.keytab
--principal=kudu/127.10.191.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:46105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:39843,127.10.191.253:42935,127.10.191.252:36993
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:16:40.306427 1393 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.191.253" port: 42935 } attempt: 1
I20260620 23:16:40.307646 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.307201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:42935 (local address 127.0.0.1:56392)
0620 23:16:40.307399 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:40.307509 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.253:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 23:16:40.308606 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.308301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:42935 (local address 127.0.0.1:56394)
0620 23:16:40.308426 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:40.308491 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.253:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 23:16:40.308808 1393 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.191.253:42935: Network error: Client connection negotiation failed: client connection to 127.10.191.253:42935: connect: Connection refused (error 111)
I20260620 23:16:40.377108 1393 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.191.253" port: 42935 } attempt: 2
I20260620 23:16:40.378439 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.377894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:42935 (local address 127.0.0.1:56400)
0620 23:16:40.378166 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:40.378294 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.253:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 23:16:40.379508 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.379161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:42935 (local address 127.0.0.1:56414)
0620 23:16:40.379305 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:40.379391 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.253:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 23:16:40.379705 1393 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.191.253:42935: Network error: Client connection negotiation failed: client connection to 127.10.191.253:42935: connect: Connection refused (error 111)
I20260620 23:16:40.470989 1393 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.191.253" port: 42935 } attempt: 3
I20260620 23:16:40.472190 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.471732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:42935 (local address 127.0.0.1:56430)
0620 23:16:40.471953 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:40.472050 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.253:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 23:16:40.473217 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.472869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:42935 (local address 127.0.0.1:56434)
0620 23:16:40.473009 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:40.473084 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.253:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 23:16:40.473417 1393 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.191.253:42935: Network error: Client connection negotiation failed: client connection to 127.10.191.253:42935: connect: Connection refused (error 111)
W20260620 23:16:40.594631 1400 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:16:40.595005 1400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:40.595129 1400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:40.595280 1400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:16:40.603478 1400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:16:40.603582 1400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:40.603678 1400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:40.603752 1400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:16:40.603842 1400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:16:40.613317 1400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:46105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:39843,127.10.191.253:42935,127.10.191.252:36993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.253.keytab
--principal=kudu/127.10.191.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.253:42935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.191.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:16:40.614550 1400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:40.615885 1400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 23:16:40.616675 1393 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.191.253" port: 42935 } attempt: 4
I20260620 23:16:40.617823 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.617308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:42935 (local address 127.0.0.1:56440)
0620 23:16:40.617607 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:40.617691 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.253:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260620 23:16:40.618799 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.618451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:42935 (local address 127.0.0.1:56454)
0620 23:16:40.618602 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:40.618682 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.253:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 23:16:40.618981 1393 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.191.253:42935: Network error: Client connection negotiation failed: client connection to 127.10.191.253:42935: connect: Connection refused (error 111)
W20260620 23:16:40.626479 1405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:40.627112 1406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:40.628203 1408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:40.628897 1400 server_base.cc:1034] running on GCE node
I20260620 23:16:40.629657 1400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:40.631006 1400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:40.632223 1400 hybrid_clock.cc:648] HybridClock initialized: now 1781997400632172 us; error 40 us; skew 500 ppm
I20260620 23:16:40.632534 1400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:16:40 dist-test-slave-jkbq krb5kdc[1313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997400, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:40.635530 1400 init.cc:377] Logged in from keytab as kudu/127.10.191.253@KRBTEST.COM (short username kudu)
I20260620 23:16:40.636965 1400 webserver.cc:460] Webserver started at http://127.10.191.253:46875/ using document root <none> and password file <none>
I20260620 23:16:40.637647 1400 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:40.637789 1400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:40.638103 1400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:40.640956 1400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-1/data/instance:
uuid: "43d86874db344e2088e4017aeea2ccfb"
format_stamp: "Formatted at 2026-06-20 23:16:40 on dist-test-slave-jkbq"
server_key: "33e48b288e9909be37d505f61f1fa762"
server_key_iv: "07b8356af18892d4e6c681e4d5d05b0e"
server_key_version: "encryptionkey@0"
I20260620 23:16:40.641721 1400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-1/wal/instance:
uuid: "43d86874db344e2088e4017aeea2ccfb"
format_stamp: "Formatted at 2026-06-20 23:16:40 on dist-test-slave-jkbq"
server_key: "33e48b288e9909be37d505f61f1fa762"
server_key_iv: "07b8356af18892d4e6c681e4d5d05b0e"
server_key_version: "encryptionkey@0"
I20260620 23:16:40.646606 1400 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 23:16:40.649834 1415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:40.651051 1400 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260620 23:16:40.651234 1400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-1/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-1/wal
uuid: "43d86874db344e2088e4017aeea2ccfb"
format_stamp: "Formatted at 2026-06-20 23:16:40 on dist-test-slave-jkbq"
server_key: "33e48b288e9909be37d505f61f1fa762"
server_key_iv: "07b8356af18892d4e6c681e4d5d05b0e"
server_key_version: "encryptionkey@0"
I20260620 23:16:40.651448 1400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:40.665416 1400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:40.669269 1400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:40.669538 1400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:40.687983 1400 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.253:42935
I20260620 23:16:40.688001 1466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.253:42935 every 8 connection(s)
I20260620 23:16:40.689286 1400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-1/data/info.pb
I20260620 23:16:40.692708 1467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:40.697753 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 1400
I20260620 23:16:40.697922 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-1/wal/instance
I20260620 23:16:40.698431 11007 external_mini_cluster.cc:1428] Setting key 19cea102a4b323941dff2fdc35358d48
I20260620 23:16:40.699563 1467 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39843 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:16:40 dist-test-slave-jkbq krb5kdc[1313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997400, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.253@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:16:40.722721 1397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.702778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43608 (local address 127.10.191.254:39843)
0620 23:16:40.703034 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:40.703043 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:40.704066 (+ 1023us) server_negotiation.cc:407] Connection header received
0620 23:16:40.706924 (+ 2858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:40.706928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:40.706998 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:40.707083 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:40.708945 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.710088 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.711611 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.712392 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.712530 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:40.715813 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:40.715823 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:40.715832 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:40.715877 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:40.719441 (+ 3564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:40.720134 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:40.720143 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:40.720150 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:40.720299 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:40.720692 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:40.720700 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:40.720706 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:40.721109 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:40.721272 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:40.721998 (+ 726us) server_negotiation.cc:299] Negotiation successful
0620 23:16:40.722277 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 23:16:40.723322 1469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.703153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39843 (local address 127.0.0.1:43608)
0620 23:16:40.703910 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:40.703962 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:40.706586 (+ 2624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:40.707335 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:40.707348 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:40.708135 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:16:40.708767 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:40.708783 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.710235 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.710239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:40.711461 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:40.711473 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.712597 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.712602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:40.712794 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:40.713453 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:16:40.713479 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:16:40.715572 (+ 2093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:16:40.719573 (+ 4001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:40.719579 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:40.719591 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:40.719997 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:40.720406 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:40.720409 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:40.720412 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:40.720572 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:40.721251 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:16:40.721262 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:16:40.721584 (+ 322us) client_negotiation.cc:769] Sending connection context
0620 23:16:40.721819 (+ 235us) client_negotiation.cc:240] Negotiation successful
0620 23:16:40.722202 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":285,"threads_started":1}
I20260620 23:16:40.725047 1467 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.191.253" port: 42935 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.10.191.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.191.252@KRBTEST.COM" created.
Jun 20 23:16:40 dist-test-slave-jkbq krb5kdc[1313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997400, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.253@KRBTEST.COM for kudu/127.10.191.253@KRBTEST.COM
I20260620 23:16:40.739995 1469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.725829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:42935 (local address 127.0.0.1:56464)
0620 23:16:40.725988 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:40.726006 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:40.726102 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:40.726946 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:40.726950 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:40.727413 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:16:40.727540 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:40.727548 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.728898 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.728907 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:40.730432 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:40.730449 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.731273 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.731285 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:40.731506 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:40.732097 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:16:40.732123 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:16:40.734505 (+ 2382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:16:40.737599 (+ 3094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:40.737604 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:40.737609 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:40.737949 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:40.738302 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:40.738308 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:40.738312 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:40.738449 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:40.739044 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:16:40.739052 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:16:40.739192 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 23:16:40.739344 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 23:16:40.739598 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 23:16:40.740096 1471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.725873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56464 (local address 127.10.191.253:42935)
0620 23:16:40.726473 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:40.726479 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:40.726504 (+ 25us) server_negotiation.cc:407] Connection header received
0620 23:16:40.726545 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:40.726551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:40.726712 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:40.726832 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:40.727688 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.728757 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.730566 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.731132 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.731226 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:40.734701 (+ 3475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:40.734711 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:40.734723 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:40.734759 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:40.737464 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:40.738075 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:40.738089 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:40.738091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:40.738195 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:40.738548 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:40.738556 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:40.738559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:40.738932 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:40.739130 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:40.739495 (+ 365us) server_negotiation.cc:299] Negotiation successful
0620 23:16:40.739700 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":445,"thread_start_us":304,"threads_started":1}
I20260620 23:16:40.742556 1467 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.191.252" port: 36993 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:16:40.743605 1469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.743183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:36993 (local address 127.0.0.1:53854)
0620 23:16:40.743334 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:40.743453 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:36993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 23:16:40.743880 1418 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.191.252:36993: connect: Connection refused (error 111)
I20260620 23:16:40.745481 1469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.745170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:36993 (local address 127.0.0.1:53856)
0620 23:16:40.745312 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:40.745383 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:36993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 23:16:40.745710 1467 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.191.252:36993: Network error: Client connection negotiation failed: client connection to 127.10.191.252:36993: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.191.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.252.keytab.
Entry for principal kudu/127.10.191.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.252.keytab.
I20260620 23:16:40.758622 11007 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
/tmp/dist-test-taskpRJWBp/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.252:36993
--webserver_interface=127.10.191.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.252.keytab
--principal=kudu/127.10.191.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.191.212:46105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.191.254:39843,127.10.191.253:42935,127.10.191.252:36993
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:16:40.795028 1467 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.191.252" port: 36993 } attempt: 1
I20260620 23:16:40.796310 1469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.795700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:36993 (local address 127.0.0.1:53860)
0620 23:16:40.795932 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:40.796073 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:36993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 23:16:40.797370 1469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.796962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:36993 (local address 127.0.0.1:53864)
0620 23:16:40.797138 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:40.797240 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:36993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 23:16:40.797605 1467 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.191.252:36993: Network error: Client connection negotiation failed: client connection to 127.10.191.252:36993: connect: Connection refused (error 111)
I20260620 23:16:40.865849 1467 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.191.252" port: 36993 } attempt: 2
I20260620 23:16:40.867215 1469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.866649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:36993 (local address 127.0.0.1:53866)
0620 23:16:40.866905 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:40.867016 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:36993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 23:16:40.868314 1469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.867911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:36993 (local address 127.0.0.1:53878)
0620 23:16:40.868078 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:40.868165 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:36993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260620 23:16:40.868577 1467 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.191.252:36993: Network error: Client connection negotiation failed: client connection to 127.10.191.252:36993: connect: Connection refused (error 111)
I20260620 23:16:40.918251 1393 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.191.253" port: 42935 } attempt: 5
Jun 20 23:16:40 dist-test-slave-jkbq krb5kdc[1313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997400, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for kudu/127.10.191.253@KRBTEST.COM
I20260620 23:16:40.932246 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.919073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:42935 (local address 127.0.0.1:56476)
0620 23:16:40.919315 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:40.919341 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:40.919474 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:40.919866 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:40.919872 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:40.920273 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:16:40.920453 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:40.920462 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.921498 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.921506 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:40.922396 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:40.922409 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.923216 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.923222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:40.923445 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:40.924105 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:16:40.924123 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:16:40.926531 (+ 2408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:16:40.929677 (+ 3146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:40.929683 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:40.929687 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:40.930062 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:40.930452 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:40.930457 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:40.930461 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:40.930609 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:40.931251 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:16:40.931257 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:16:40.931423 (+ 166us) client_negotiation.cc:769] Sending connection context
0620 23:16:40.931570 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 23:16:40.931870 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111}
I20260620 23:16:40.932245 1471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.919159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56476 (local address 127.10.191.253:42935)
0620 23:16:40.919456 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:40.919462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:40.919484 (+ 22us) server_negotiation.cc:407] Connection header received
0620 23:16:40.919622 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:40.919628 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:40.919701 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:40.919816 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:40.920621 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.921328 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.922554 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:40.923044 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:40.923201 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:40.926739 (+ 3538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:40.926751 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:40.926755 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:40.926788 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:40.929510 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:40.930191 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:40.930198 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:40.930202 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:40.930335 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:40.930728 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:40.930734 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:40.930737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:40.931113 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:40.931290 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:40.931690 (+ 400us) server_negotiation.cc:299] Negotiation successful
0620 23:16:40.931908 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171}
I20260620 23:16:40.933398 1393 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.191.252" port: 36993 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:16:40.934381 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.934003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:36993 (local address 127.0.0.1:53894)
0620 23:16:40.934163 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:40.934245 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:36993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 23:16:40.935326 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.935012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:36993 (local address 127.0.0.1:53896)
0620 23:16:40.935145 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:40.935234 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:36993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 23:16:40.935513 1393 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.191.252:36993: Network error: Client connection negotiation failed: client connection to 127.10.191.252:36993: connect: Connection refused (error 111)
I20260620 23:16:40.959825 1467 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.191.252" port: 36993 } attempt: 3
I20260620 23:16:40.961040 1469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.960530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:36993 (local address 127.0.0.1:53912)
0620 23:16:40.960785 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:40.960881 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:36993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260620 23:16:40.962283 1469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.961839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:36993 (local address 127.0.0.1:53928)
0620 23:16:40.962090 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:40.962174 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:36993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260620 23:16:40.962496 1467 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.191.252:36993: Network error: Client connection negotiation failed: client connection to 127.10.191.252:36993: connect: Connection refused (error 111)
I20260620 23:16:40.986730 1393 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.191.252" port: 36993 } attempt: 1
I20260620 23:16:40.988010 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.987541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:36993 (local address 127.0.0.1:53942)
0620 23:16:40.987749 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:40.987842 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:36993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 23:16:40.989027 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:40.988687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:36993 (local address 127.0.0.1:53948)
0620 23:16:40.988851 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:40.988918 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:36993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 23:16:40.989270 1393 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.191.252:36993: Network error: Client connection negotiation failed: client connection to 127.10.191.252:36993: connect: Connection refused (error 111)
I20260620 23:16:41.057497 1393 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.191.252" port: 36993 } attempt: 2
I20260620 23:16:41.058710 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:41.058195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:36993 (local address 127.0.0.1:53964)
0620 23:16:41.058437 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:41.058528 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:36993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 23:16:41.059758 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:41.059394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:36993 (local address 127.0.0.1:53980)
0620 23:16:41.059553 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:41.059642 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:36993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 23:16:41.059978 1393 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.191.252:36993: Network error: Client connection negotiation failed: client connection to 127.10.191.252:36993: connect: Connection refused (error 111)
W20260620 23:16:41.103108 1474 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:16:41.103514 1474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:16:41.103647 1474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:16:41.103801 1474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 23:16:41.105695 1467 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.191.252" port: 36993 } attempt: 4
I20260620 23:16:41.106768 1469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:41.106299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:36993 (local address 127.0.0.1:53996)
0620 23:16:41.106532 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:41.106611 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:36993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 23:16:41.107780 1469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:41.107471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:36993 (local address 127.0.0.1:54006)
0620 23:16:41.107624 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:41.107684 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:36993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 23:16:41.107976 1467 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.191.252:36993: Network error: Client connection negotiation failed: client connection to 127.10.191.252:36993: connect: Connection refused (error 111)
W20260620 23:16:41.111450 1474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:16:41.111558 1474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:16:41.111692 1474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:16:41.111780 1474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:16:41.111889 1474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:16:41.121731 1474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.191.212:46105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.191.254:39843,127.10.191.253:42935,127.10.191.252:36993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/krb5kdc/kudu_127.10.191.252.keytab
--principal=kudu/127.10.191.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.191.252:36993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.191.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:16:41.123107 1474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:16:41.124387 1474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:16:41.136013 1479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:41.135980 1482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:16:41.136112 1480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:16:41.136899 1474 server_base.cc:1034] running on GCE node
I20260620 23:16:41.137645 1474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:16:41.139034 1474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:16:41.140251 1474 hybrid_clock.cc:648] HybridClock initialized: now 1781997401140190 us; error 34 us; skew 500 ppm
I20260620 23:16:41.140589 1474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:16:41 dist-test-slave-jkbq krb5kdc[1313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997401, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:16:41.143683 1474 init.cc:377] Logged in from keytab as kudu/127.10.191.252@KRBTEST.COM (short username kudu)
I20260620 23:16:41.145339 1474 webserver.cc:460] Webserver started at http://127.10.191.252:39759/ using document root <none> and password file <none>
I20260620 23:16:41.146014 1474 fs_manager.cc:359] Metadata directory not provided
I20260620 23:16:41.146147 1474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:16:41.146458 1474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:16:41.149355 1474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-2/data/instance:
uuid: "8046495c98464d89b87137568e1da7a7"
format_stamp: "Formatted at 2026-06-20 23:16:41 on dist-test-slave-jkbq"
server_key: "fb7784c8af081969f541132befdda3cd"
server_key_iv: "6e6d7061a95d1193b8bf8a354df5fb40"
server_key_version: "encryptionkey@0"
I20260620 23:16:41.150121 1474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-2/wal/instance:
uuid: "8046495c98464d89b87137568e1da7a7"
format_stamp: "Formatted at 2026-06-20 23:16:41 on dist-test-slave-jkbq"
server_key: "fb7784c8af081969f541132befdda3cd"
server_key_iv: "6e6d7061a95d1193b8bf8a354df5fb40"
server_key_version: "encryptionkey@0"
I20260620 23:16:41.155125 1474 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 23:16:41.158511 1489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:16:41.159970 1474 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:16:41.160151 1474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-2/data,/tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-2/wal
uuid: "8046495c98464d89b87137568e1da7a7"
format_stamp: "Formatted at 2026-06-20 23:16:41 on dist-test-slave-jkbq"
server_key: "fb7784c8af081969f541132befdda3cd"
server_key_iv: "6e6d7061a95d1193b8bf8a354df5fb40"
server_key_version: "encryptionkey@0"
I20260620 23:16:41.160373 1474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:16:41.166177 1393 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.191.252" port: 36993 } attempt: 3
I20260620 23:16:41.167307 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:41.166774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:36993 (local address 127.0.0.1:54014)
0620 23:16:41.167067 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:41.167153 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:36993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 23:16:41.168345 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:41.167981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:36993 (local address 127.0.0.1:54020)
0620 23:16:41.168152 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:41.168236 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.191.252:36993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 23:16:41.168587 1393 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.191.252:36993: Network error: Client connection negotiation failed: client connection to 127.10.191.252:36993: connect: Connection refused (error 111)
I20260620 23:16:41.185967 1474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:16:41.190156 1474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:16:41.190434 1474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:16:41.208629 1474 rpc_server.cc:307] RPC server started. Bound to: 127.10.191.252:36993
I20260620 23:16:41.208643 1540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.191.252:36993 every 8 connection(s)
I20260620 23:16:41.209956 1474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-2/data/info.pb
I20260620 23:16:41.212208 11007 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu as pid 1474
I20260620 23:16:41.212425 11007 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpRJWBp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781997042893763-11007-0/minicluster-data/master-2/wal/instance
I20260620 23:16:41.212850 11007 external_mini_cluster.cc:1428] Setting key d15daee285223343df6b3901c5f789e7
I20260620 23:16:41.213956 1541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:16:41.224361 1541 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39843 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:16:41 dist-test-slave-jkbq krb5kdc[1313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.252@KRBTEST.COM
I20260620 23:16:41.235355 1544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:41.214632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54024 (local address 127.10.191.252:36993)
0620 23:16:41.215523 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:41.215535 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:41.215572 (+ 37us) server_negotiation.cc:407] Connection header received
0620 23:16:41.217087 (+ 1515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:41.217114 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:41.217484 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:41.217869 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:41.219284 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:41.220560 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:41.221790 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:41.222447 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:41.222640 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:41.226913 (+ 4273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:41.226930 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:41.226947 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:41.226991 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:41.230975 (+ 3984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:41.231836 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:41.231842 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:41.231847 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:41.232029 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:41.232452 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:41.232456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:41.232458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:41.233110 (+ 652us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:41.233433 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:41.233817 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 23:16:41.234234 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":442,"threads_started":1}
Jun 20 23:16:41 dist-test-slave-jkbq krb5kdc[1313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997401, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.252@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
I20260620 23:16:41.243173 1545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:41.227043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43612 (local address 127.10.191.254:39843)
0620 23:16:41.227686 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:41.227693 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:41.228216 (+ 523us) server_negotiation.cc:407] Connection header received
0620 23:16:41.228428 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:41.228436 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:41.228519 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:41.228646 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:41.229796 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:41.230975 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:41.232780 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:41.233416 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:41.233569 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:41.236778 (+ 3209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:41.236785 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:41.236791 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:41.236832 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:41.240376 (+ 3544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:41.241034 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:41.241040 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:41.241044 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:41.241221 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:41.241621 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:41.241627 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:41.241630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:41.241902 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:41.242069 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:41.242533 (+ 464us) server_negotiation.cc:299] Negotiation successful
0620 23:16:41.242745 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":351,"threads_started":1}
I20260620 23:16:41.243173 1546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:41.227095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.254:39843 (local address 127.0.0.1:43612)
0620 23:16:41.228065 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:41.228108 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:41.228267 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:41.228727 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:41.228732 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:41.229361 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:16:41.229644 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:41.229653 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:41.231140 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:41.231145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:41.232644 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:41.232656 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:41.233619 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:41.233626 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:41.233825 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:41.234452 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:16:41.234468 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:16:41.236591 (+ 2123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:16:41.240503 (+ 3912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:41.240512 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:41.240516 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:41.240906 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:41.241355 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:41.241362 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:41.241367 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:41.241503 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:41.242036 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:16:41.242042 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:16:41.242245 (+ 203us) client_negotiation.cc:769] Sending connection context
0620 23:16:41.242414 (+ 169us) client_negotiation.cc:240] Negotiation successful
0620 23:16:41.242718 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"thread_start_us":588,"threads_started":1}
I20260620 23:16:41.245096 1541 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.191.253" port: 42935 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:16:41 dist-test-slave-jkbq krb5kdc[1313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.254@KRBTEST.COM
Jun 20 23:16:41 dist-test-slave-jkbq krb5kdc[1313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997401, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.252@KRBTEST.COM for kudu/127.10.191.253@KRBTEST.COM
I20260620 23:16:41.254520 1547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:41.238462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43624 (local address 127.10.191.254:39843)
0620 23:16:41.239416 (+ 954us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:41.239420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:41.239439 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:16:41.239481 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:41.239484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:41.239534 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:41.239614 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:41.241178 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:41.242021 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:41.243377 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:41.244103 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:41.244246 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:41.248455 (+ 4209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:41.248461 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:41.248467 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:41.248501 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:41.251771 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:41.252597 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:41.252600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:41.252602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:41.252769 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:41.253137 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:41.253140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:41.253142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:41.253378 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:41.253629 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:41.253929 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 23:16:41.254132 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":835,"thread_start_us":633,"threads_started":1}
I20260620 23:16:41.259239 1546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:41.245826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.253:42935 (local address 127.0.0.1:56478)
0620 23:16:41.245973 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:41.245993 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:41.246084 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:41.246393 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:41.246398 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:41.246751 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:16:41.246953 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:41.246963 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:41.247952 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:41.247958 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:41.249215 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:41.249226 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:41.250060 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:41.250065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:41.250294 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:41.250779 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:16:41.250801 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:16:41.253314 (+ 2513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:16:41.256978 (+ 3664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:41.256981 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:41.256984 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:41.257506 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:41.257847 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:41.257854 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:41.257859 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:41.258032 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:41.258526 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:16:41.258532 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:16:41.258670 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 23:16:41.258779 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 23:16:41.259021 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 23:16:41.259538 1471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:41.245863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56478 (local address 127.10.191.253:42935)
0620 23:16:41.246122 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:41.246130 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:41.246148 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:16:41.246204 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:41.246209 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:41.246274 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:41.246374 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:41.247095 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:41.247824 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:41.249395 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:41.249877 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:41.250046 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:41.253495 (+ 3449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:41.253499 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:41.253502 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:41.253531 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:41.256858 (+ 3327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:41.257619 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:41.257623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:41.257625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:41.257735 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:41.258162 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:41.258165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:41.258167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:41.258406 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:41.258559 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:41.258947 (+ 388us) server_negotiation.cc:299] Negotiation successful
0620 23:16:41.259168 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260620 23:16:41.260375 1541 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.191.252" port: 36993 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:16:41 dist-test-slave-jkbq krb5kdc[1313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.191.253@KRBTEST.COM
Jun 20 23:16:41 dist-test-slave-jkbq krb5kdc[1313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997401, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.252@KRBTEST.COM for kudu/127.10.191.252@KRBTEST.COM
I20260620 23:16:41.271886 1548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:41.256237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56494 (local address 127.10.191.253:42935)
0620 23:16:41.256850 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:41.256855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:41.256878 (+ 23us) server_negotiation.cc:407] Connection header received
0620 23:16:41.256924 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:41.256930 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:41.257008 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:41.257095 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:41.258545 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:41.259713 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:41.261192 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:41.261815 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:41.261944 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:41.266271 (+ 4327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:41.266278 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:41.266284 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:41.266320 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:41.269170 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:41.269974 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:41.269978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:41.269980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:41.270101 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:41.270491 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:41.270494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:41.270496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:41.270733 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:41.271002 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:41.271290 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 23:16:41.271500 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":259,"threads_started":1}
I20260620 23:16:41.273658 1546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:41.260968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:36993 (local address 127.0.0.1:54034)
0620 23:16:41.261075 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:41.261088 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:41.261215 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:41.261564 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:41.261569 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:41.262036 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:16:41.262186 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:41.262194 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:41.263185 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:41.263192 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:41.264135 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:41.264146 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:41.264875 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:41.264881 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:41.265098 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:41.265632 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:16:41.265648 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:16:41.267813 (+ 2165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:16:41.271005 (+ 3192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:41.271009 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:41.271014 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:41.271482 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:41.271901 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:41.271908 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:41.271912 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:41.272066 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:41.272648 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:16:41.272652 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:16:41.272820 (+ 168us) client_negotiation.cc:769] Sending connection context
0620 23:16:41.272980 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 23:16:41.273255 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 23:16:41.273759 1544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:41.261171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54034 (local address 127.10.191.252:36993)
0620 23:16:41.261347 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:41.261351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:41.261364 (+ 13us) server_negotiation.cc:407] Connection header received
0620 23:16:41.261398 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:41.261401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:41.261446 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:41.261514 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:41.262312 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:41.263067 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:41.264265 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:41.264708 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:41.264860 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:41.268011 (+ 3151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:41.268019 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:41.268025 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:41.268062 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:41.270825 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:41.271641 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:41.271647 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:41.271650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:41.271788 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:41.272193 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:41.272197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:41.272200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:41.272514 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:41.272654 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:41.273053 (+ 399us) server_negotiation.cc:299] Negotiation successful
0620 23:16:41.273408 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":132,"server-negotiator.queue_time_us":74}
I20260620 23:16:41.276768 11007 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:16:41.278849 1541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8046495c98464d89b87137568e1da7a7: Bootstrap starting.
I20260620 23:16:41.279141 1368 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "158c6280f017418b9911d4359f39fa1b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43624
I20260620 23:16:41.282119 1541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8046495c98464d89b87137568e1da7a7: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:41.283181 1541 log.cc:824] T 00000000000000000000000000000000 P 8046495c98464d89b87137568e1da7a7: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:41.285866 1541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8046495c98464d89b87137568e1da7a7: No bootstrap required, opened a new log
I20260620 23:16:41.292609 1541 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8046495c98464d89b87137568e1da7a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158c6280f017418b9911d4359f39fa1b" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39843 } } peers { permanent_uuid: "43d86874db344e2088e4017aeea2ccfb" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 42935 } } peers { permanent_uuid: "8046495c98464d89b87137568e1da7a7" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 36993 } }
I20260620 23:16:41.292951 1541 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8046495c98464d89b87137568e1da7a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8046495c98464d89b87137568e1da7a7, State: Initialized, Role: FOLLOWER
I20260620 23:16:41.293368 1541 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8046495c98464d89b87137568e1da7a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158c6280f017418b9911d4359f39fa1b" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39843 } } peers { permanent_uuid: "43d86874db344e2088e4017aeea2ccfb" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 42935 } } peers { permanent_uuid: "8046495c98464d89b87137568e1da7a7" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 36993 } }
I20260620 23:16:41.294260 1550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8046495c98464d89b87137568e1da7a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158c6280f017418b9911d4359f39fa1b" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39843 } } peers { permanent_uuid: "43d86874db344e2088e4017aeea2ccfb" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 42935 } } peers { permanent_uuid: "8046495c98464d89b87137568e1da7a7" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 36993 } } }
I20260620 23:16:41.294632 1550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8046495c98464d89b87137568e1da7a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:16:41.295199 1541 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8046495c98464d89b87137568e1da7a7 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 23:16:41.304272 1561 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8046495c98464d89b87137568e1da7a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:16:41.304400 1561 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:16:41.302349 1474 master_runner.cc:186] Error getting master registration for 127.10.191.254:39843: OK, instance_id { permanent_uuid: "158c6280f017418b9911d4359f39fa1b" instance_seqno: 1781997400210535 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 23:16:41.310177 1474 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 23:16:41.345808 1393 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.191.252" port: 36993 } attempt: 4
Jun 20 23:16:41 dist-test-slave-jkbq krb5kdc[1313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997400, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.254@KRBTEST.COM for kudu/127.10.191.252@KRBTEST.COM
I20260620 23:16:41.363936 1544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:41.346812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54038 (local address 127.10.191.252:36993)
0620 23:16:41.347058 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:41.347064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:41.347104 (+ 40us) server_negotiation.cc:407] Connection header received
0620 23:16:41.347278 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:41.347285 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:41.347377 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:41.347489 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:41.348542 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:41.349679 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:41.351737 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:41.352403 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:41.352533 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:41.356693 (+ 4160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:41.356700 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:41.356705 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:41.356744 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:41.360606 (+ 3862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:41.361477 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:41.361484 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:41.361488 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:41.361642 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:41.362112 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:41.362119 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:41.362125 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:41.362611 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:41.362761 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:41.363301 (+ 540us) server_negotiation.cc:299] Negotiation successful
0620 23:16:41.363561 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260620 23:16:41.363942 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:41.346679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:36993 (local address 127.0.0.1:54038)
0620 23:16:41.346914 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:41.346939 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:41.347083 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:41.347544 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:41.347552 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:41.348128 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:16:41.348363 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:41.348376 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:41.349820 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:41.349826 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:41.351559 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:41.351586 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:41.352567 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:41.352582 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:41.352796 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:41.353541 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:16:41.353577 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:16:41.356491 (+ 2914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:16:41.360760 (+ 4269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:41.360766 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:41.360771 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:41.361334 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:41.361772 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:41.361779 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:41.361784 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:41.361973 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:41.362738 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:16:41.362743 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:16:41.362992 (+ 249us) client_negotiation.cc:769] Sending connection context
0620 23:16:41.363150 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 23:16:41.363467 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 23:16:41.368978 1393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 158c6280f017418b9911d4359f39fa1b: Bootstrap starting.
I20260620 23:16:41.373488 1393 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 158c6280f017418b9911d4359f39fa1b: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:41.374933 1393 log.cc:824] T 00000000000000000000000000000000 P 158c6280f017418b9911d4359f39fa1b: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:41.378458 1393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 158c6280f017418b9911d4359f39fa1b: No bootstrap required, opened a new log
I20260620 23:16:41.386808 1393 raft_consensus.cc:348] T 00000000000000000000000000000000 P 158c6280f017418b9911d4359f39fa1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158c6280f017418b9911d4359f39fa1b" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39843 } } peers { permanent_uuid: "43d86874db344e2088e4017aeea2ccfb" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 42935 } } peers { permanent_uuid: "8046495c98464d89b87137568e1da7a7" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 36993 } }
I20260620 23:16:41.387264 1393 raft_consensus.cc:729] T 00000000000000000000000000000000 P 158c6280f017418b9911d4359f39fa1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 158c6280f017418b9911d4359f39fa1b, State: Initialized, Role: FOLLOWER
I20260620 23:16:41.387753 1393 consensus_queue.cc:260] T 00000000000000000000000000000000 P 158c6280f017418b9911d4359f39fa1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158c6280f017418b9911d4359f39fa1b" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39843 } } peers { permanent_uuid: "43d86874db344e2088e4017aeea2ccfb" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 42935 } } peers { permanent_uuid: "8046495c98464d89b87137568e1da7a7" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 36993 } }
I20260620 23:16:41.388758 1564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 158c6280f017418b9911d4359f39fa1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158c6280f017418b9911d4359f39fa1b" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39843 } } peers { permanent_uuid: "43d86874db344e2088e4017aeea2ccfb" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 42935 } } peers { permanent_uuid: "8046495c98464d89b87137568e1da7a7" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 36993 } } }
I20260620 23:16:41.389243 1564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 158c6280f017418b9911d4359f39fa1b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:16:41.390004 1393 sys_catalog.cc:564] T 00000000000000000000000000000000 P 158c6280f017418b9911d4359f39fa1b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:16:41.390156 1368 raft_consensus.cc:482] T 00000000000000000000000000000000 P 158c6280f017418b9911d4359f39fa1b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 23:16:41.390377 1368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 158c6280f017418b9911d4359f39fa1b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:41.393844 1368 raft_consensus.cc:504] T 00000000000000000000000000000000 P 158c6280f017418b9911d4359f39fa1b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158c6280f017418b9911d4359f39fa1b" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39843 } } peers { permanent_uuid: "43d86874db344e2088e4017aeea2ccfb" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 42935 } } peers { permanent_uuid: "8046495c98464d89b87137568e1da7a7" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 36993 } }
I20260620 23:16:41.395407 1368 leader_election.cc:288] T 00000000000000000000000000000000 P 158c6280f017418b9911d4359f39fa1b [CANDIDATE]: Term 1 election: Requested vote from peers 43d86874db344e2088e4017aeea2ccfb (127.10.191.253:42935), 8046495c98464d89b87137568e1da7a7 (127.10.191.252:36993)
I20260620 23:16:41.396703 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "158c6280f017418b9911d4359f39fa1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8046495c98464d89b87137568e1da7a7"
I20260620 23:16:41.397360 1516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8046495c98464d89b87137568e1da7a7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:41.402315 1516 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8046495c98464d89b87137568e1da7a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 158c6280f017418b9911d4359f39fa1b in term 1.
I20260620 23:16:41.403378 1344 leader_election.cc:302] T 00000000000000000000000000000000 P 158c6280f017418b9911d4359f39fa1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 158c6280f017418b9911d4359f39fa1b, 8046495c98464d89b87137568e1da7a7; no voters:
I20260620 23:16:41.404016 1564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 158c6280f017418b9911d4359f39fa1b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:16:41.407153 1467 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.191.252" port: 36993 } attempt: 5
I20260620 23:16:41.411405 1564 raft_consensus.cc:686] T 00000000000000000000000000000000 P 158c6280f017418b9911d4359f39fa1b [term 1 LEADER]: Becoming Leader. State: Replica: 158c6280f017418b9911d4359f39fa1b, State: Running, Role: LEADER
I20260620 23:16:41.412319 1564 consensus_queue.cc:237] T 00000000000000000000000000000000 P 158c6280f017418b9911d4359f39fa1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158c6280f017418b9911d4359f39fa1b" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39843 } } peers { permanent_uuid: "43d86874db344e2088e4017aeea2ccfb" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 42935 } } peers { permanent_uuid: "8046495c98464d89b87137568e1da7a7" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 36993 } }
Jun 20 23:16:41 dist-test-slave-jkbq krb5kdc[1313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781997400, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.191.253@KRBTEST.COM for kudu/127.10.191.252@KRBTEST.COM
I20260620 23:16:41.429840 1571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 158c6280f017418b9911d4359f39fa1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 158c6280f017418b9911d4359f39fa1b. Latest consensus state: current_term: 1 leader_uuid: "158c6280f017418b9911d4359f39fa1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158c6280f017418b9911d4359f39fa1b" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39843 } } peers { permanent_uuid: "43d86874db344e2088e4017aeea2ccfb" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 42935 } } peers { permanent_uuid: "8046495c98464d89b87137568e1da7a7" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 36993 } } }
I20260620 23:16:41.430548 1571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 158c6280f017418b9911d4359f39fa1b [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:41.438607 1581 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260620 23:16:41.442775 1495 tablet.cc:2369] T 00000000000000000000000000000000 P 8046495c98464d89b87137568e1da7a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:16:41.445361 1469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:41.407859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.191.252:36993 (local address 127.0.0.1:54040)
0620 23:16:41.408288 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0620 23:16:41.408312 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:16:41.408456 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:16:41.410082 (+ 1626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:16:41.410092 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:16:41.410833 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:16:41.411157 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:41.411175 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:41.418237 (+ 7062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:41.418248 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:41.420008 (+ 1760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:16:41.420029 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:16:41.421164 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:16:41.421175 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:16:41.421536 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:41.422519 (+ 983us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:16:41.422556 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:16:41.426640 (+ 4084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:16:41.440972 (+ 14332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:41.440982 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:41.440991 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:41.441774 (+ 783us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:41.442500 (+ 726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:16:41.442509 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:16:41.442517 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:16:41.442818 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:16:41.443834 (+ 1016us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:16:41.443843 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:16:41.444074 (+ 231us) client_negotiation.cc:769] Sending connection context
0620 23:16:41.444266 (+ 192us) client_negotiation.cc:240] Negotiation successful
0620 23:16:41.444745 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271}
I20260620 23:16:41.447207 1544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:41.408273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54040 (local address 127.10.191.252:36993)
0620 23:16:41.409054 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:41.409064 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:41.409091 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:16:41.409171 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:41.409179 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:41.409315 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:41.415995 (+ 6680us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:41.416257 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:41.417985 (+ 1728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:41.420256 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:41.420948 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:41.426923 (+ 5975us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:41.427227 (+ 304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:41.427237 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:41.427245 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:41.427291 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:41.440709 (+ 13418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:41.442022 (+ 1313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:41.442032 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:41.442039 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:41.442308 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:41.443122 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:41.443131 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:41.443137 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:41.443642 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:41.446043 (+ 2401us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:41.446223 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 23:16:41.446569 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599}
W20260620 23:16:41.450310 1347 tablet.cc:2369] T 00000000000000000000000000000000 P 158c6280f017418b9911d4359f39fa1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:16:41.454003 1467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43d86874db344e2088e4017aeea2ccfb: Bootstrap starting.
I20260620 23:16:41.457439 1581 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:16:41.457973 1467 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43d86874db344e2088e4017aeea2ccfb: Neither blocks nor log segments found. Creating new log.
I20260620 23:16:41.459295 1467 log.cc:824] T 00000000000000000000000000000000 P 43d86874db344e2088e4017aeea2ccfb: Log is configured to *not* fsync() on all Append() calls
I20260620 23:16:41.479364 1516 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8046495c98464d89b87137568e1da7a7 [term 1 FOLLOWER]: Refusing update from remote peer 158c6280f017418b9911d4359f39fa1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:16:41.481006 1571 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 158c6280f017418b9911d4359f39fa1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8046495c98464d89b87137568e1da7a7" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 36993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:16:41.492736 1467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43d86874db344e2088e4017aeea2ccfb: No bootstrap required, opened a new log
I20260620 23:16:41.500402 1547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:41.425412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43628 (local address 127.10.191.254:39843)
0620 23:16:41.425587 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:41.425592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:41.462956 (+ 37364us) server_negotiation.cc:407] Connection header received
0620 23:16:41.465203 (+ 2247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:41.465209 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:41.465295 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:41.465408 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:41.473028 (+ 7620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:41.474209 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:41.476057 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:41.476790 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:41.476893 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:41.483183 (+ 6290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:41.483190 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:41.483195 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:41.483250 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:41.495879 (+ 12629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:41.497712 (+ 1833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:41.497720 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:41.497726 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:41.497897 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:41.498500 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:41.498506 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:41.498509 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:41.498933 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:41.499179 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:41.499761 (+ 582us) server_negotiation.cc:299] Negotiation successful
0620 23:16:41.500006 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 23:16:41.513682 1548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:41.430178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56508 (local address 127.10.191.253:42935)
0620 23:16:41.430357 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:41.430363 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:41.465914 (+ 35551us) server_negotiation.cc:407] Connection header received
0620 23:16:41.466683 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:41.466689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:41.466778 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:41.466891 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:41.468979 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:41.470030 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:41.474239 (+ 4209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:41.475217 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:41.475358 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:41.487077 (+ 11719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:41.487084 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:41.487089 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:41.487132 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:41.490617 (+ 3485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:41.509077 (+ 18460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:41.509083 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:41.509086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:41.509259 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:41.509709 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:41.509714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:41.509718 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:41.510079 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:41.510345 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:41.513058 (+ 2713us) server_negotiation.cc:299] Negotiation successful
0620 23:16:41.513334 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 23:16:41.522406 1467 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43d86874db344e2088e4017aeea2ccfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158c6280f017418b9911d4359f39fa1b" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39843 } } peers { permanent_uuid: "43d86874db344e2088e4017aeea2ccfb" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 42935 } } peers { permanent_uuid: "8046495c98464d89b87137568e1da7a7" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 36993 } }
I20260620 23:16:41.526127 1544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:41.441294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54046 (local address 127.10.191.252:36993)
0620 23:16:41.448661 (+ 7367us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:41.448670 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:41.466447 (+ 17777us) server_negotiation.cc:407] Connection header received
0620 23:16:41.472571 (+ 6124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:41.472578 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:41.472651 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:41.472721 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:41.479149 (+ 6428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:41.480285 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:41.507102 (+ 26817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:41.507935 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:41.508045 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:41.515992 (+ 7947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:16:41.515999 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:16:41.516006 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:16:41.516052 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:16:41.519945 (+ 3893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:41.523620 (+ 3675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:41.523625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:41.523629 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:41.523775 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:16:41.524256 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:16:41.524262 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:16:41.524268 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:16:41.524695 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:16:41.524931 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:41.525441 (+ 510us) server_negotiation.cc:299] Negotiation successful
0620 23:16:41.525667 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7170,"thread_start_us":20574,"threads_started":1}
I20260620 23:16:41.522832 1467 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43d86874db344e2088e4017aeea2ccfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43d86874db344e2088e4017aeea2ccfb, State: Initialized, Role: FOLLOWER
I20260620 23:16:41.527942 1467 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43d86874db344e2088e4017aeea2ccfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158c6280f017418b9911d4359f39fa1b" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39843 } } peers { permanent_uuid: "43d86874db344e2088e4017aeea2ccfb" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 42935 } } peers { permanent_uuid: "8046495c98464d89b87137568e1da7a7" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 36993 } }
I20260620 23:16:41.531873 1591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43d86874db344e2088e4017aeea2ccfb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158c6280f017418b9911d4359f39fa1b" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39843 } } peers { permanent_uuid: "43d86874db344e2088e4017aeea2ccfb" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 42935 } } peers { permanent_uuid: "8046495c98464d89b87137568e1da7a7" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 36993 } } }
I20260620 23:16:41.532974 1591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43d86874db344e2088e4017aeea2ccfb [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:16:41.533972 1467 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43d86874db344e2088e4017aeea2ccfb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:16:41.534518 1441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43d86874db344e2088e4017aeea2ccfb [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:16:41.538506 1441 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 43d86874db344e2088e4017aeea2ccfb [term 1 FOLLOWER]: Refusing update from remote peer 158c6280f017418b9911d4359f39fa1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:16:41.539119 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "158c6280f017418b9911d4359f39fa1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "43d86874db344e2088e4017aeea2ccfb"
I20260620 23:16:41.540017 1586 mvcc.cc:205] Tried to move back new op lower bound from 7299061356402585600 to 7299061356193546240. Current Snapshot: MvccSnapshot[applied={T|T < 7299061356402585600}]
I20260620 23:16:41.541023 1564 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 158c6280f017418b9911d4359f39fa1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "43d86874db344e2088e4017aeea2ccfb" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 42935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:16:41.543171 1442 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 43d86874db344e2088e4017aeea2ccfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 158c6280f017418b9911d4359f39fa1b in term 1.
I20260620 23:16:41.559351 1564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 158c6280f017418b9911d4359f39fa1b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "158c6280f017418b9911d4359f39fa1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158c6280f017418b9911d4359f39fa1b" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39843 } } peers { permanent_uuid: "43d86874db344e2088e4017aeea2ccfb" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 42935 } } peers { permanent_uuid: "8046495c98464d89b87137568e1da7a7" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 36993 } } }
I20260620 23:16:41.559654 1564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 158c6280f017418b9911d4359f39fa1b [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:41.562688 1581 catalog_manager.cc:1347] Generated new cluster ID: 7df7cc41ccf44feca7728500695ced38
I20260620 23:16:41.562803 1581 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:16:41.570156 1550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8046495c98464d89b87137568e1da7a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 158c6280f017418b9911d4359f39fa1b. Latest consensus state: current_term: 1 leader_uuid: "158c6280f017418b9911d4359f39fa1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158c6280f017418b9911d4359f39fa1b" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39843 } } peers { permanent_uuid: "43d86874db344e2088e4017aeea2ccfb" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 42935 } } peers { permanent_uuid: "8046495c98464d89b87137568e1da7a7" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 36993 } } }
I20260620 23:16:41.570524 1550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8046495c98464d89b87137568e1da7a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:16:41.572340 1595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8046495c98464d89b87137568e1da7a7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "158c6280f017418b9911d4359f39fa1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158c6280f017418b9911d4359f39fa1b" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39843 } } peers { permanent_uuid: "43d86874db344e2088e4017aeea2ccfb" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 42935 } } peers { permanent_uuid: "8046495c98464d89b87137568e1da7a7" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 36993 } } }
I20260620 23:16:41.572827 1595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8046495c98464d89b87137568e1da7a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:16:41.535145 1326 master_runner.cc:186] Error getting master registration for 127.10.191.253:42935: OK, instance_id { permanent_uuid: "43d86874db344e2088e4017aeea2ccfb" instance_seqno: 1781997400682820 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 23:16:41.576248 1326 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 23:16:41.586287 1581 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:16:41.588023 1581 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:16:41.600286 1571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 158c6280f017418b9911d4359f39fa1b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "158c6280f017418b9911d4359f39fa1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158c6280f017418b9911d4359f39fa1b" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39843 } } peers { permanent_uuid: "43d86874db344e2088e4017aeea2ccfb" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 42935 } } peers { permanent_uuid: "8046495c98464d89b87137568e1da7a7" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 36993 } } }
I20260620 23:16:41.600617 1571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 158c6280f017418b9911d4359f39fa1b [sys.catalog]: This master's current role is: LEADER
I20260620 23:16:41.600410 1591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43d86874db344e2088e4017aeea2ccfb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 158c6280f017418b9911d4359f39fa1b. Latest consensus state: current_term: 1 leader_uuid: "158c6280f017418b9911d4359f39fa1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158c6280f017418b9911d4359f39fa1b" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39843 } } peers { permanent_uuid: "43d86874db344e2088e4017aeea2ccfb" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 42935 } } peers { permanent_uuid: "8046495c98464d89b87137568e1da7a7" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 36993 } } }
I20260620 23:16:41.600725 1591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43d86874db344e2088e4017aeea2ccfb [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:16:41.601567 1601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43d86874db344e2088e4017aeea2ccfb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "158c6280f017418b9911d4359f39fa1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158c6280f017418b9911d4359f39fa1b" member_type: VOTER last_known_addr { host: "127.10.191.254" port: 39843 } } peers { permanent_uuid: "43d86874db344e2088e4017aeea2ccfb" member_type: VOTER last_known_addr { host: "127.10.191.253" port: 42935 } } peers { permanent_uuid: "8046495c98464d89b87137568e1da7a7" member_type: VOTER last_known_addr { host: "127.10.191.252" port: 36993 } } }
I20260620 23:16:41.602613 1601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43d86874db344e2088e4017aeea2ccfb [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:16:41.602989 1581 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 158c6280f017418b9911d4359f39fa1b: Generated new TSK 0
I20260620 23:16:41.603947 1581 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 23:16:41.733999 1547 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:16:41.704392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43634 (local address 127.10.191.254:39843)
0620 23:16:41.704577 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:41.704583 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:41.730141 (+ 25558us) server_negotiation.cc:407] Connection header received
0620 23:16:41.732285 (+ 2144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:41.732291 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:41.732407 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:41.732485 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:41.733713 (+ 1228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43634: BlockingRecv error: recv got EOF from 127.0.0.1:43634 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 23:16:41.736749 1544 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:16:41.730651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54060 (local address 127.10.191.252:36993)
0620 23:16:41.730832 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:41.730837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:41.735067 (+ 4230us) server_negotiation.cc:407] Connection header received
0620 23:16:41.735312 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:41.735317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:41.735391 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:41.735460 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:41.736525 (+ 1065us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54060: BlockingRecv error: recv got EOF from 127.0.0.1:54060 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 23:16:41.737013 1548 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:16:41.710717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56518 (local address 127.10.191.253:42935)
0620 23:16:41.710925 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:41.710931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:41.735069 (+ 24138us) server_negotiation.cc:407] Connection header received
0620 23:16:41.735624 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:41.735629 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:41.735715 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:41.735792 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:41.736763 (+ 971us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56518: BlockingRecv error: recv got EOF from 127.0.0.1:56518 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:16:41.741087 1618 catalog_manager.cc:1259] Loaded cluster ID: 7df7cc41ccf44feca7728500695ced38
I20260620 23:16:41.741204 1618 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 43d86874db344e2088e4017aeea2ccfb: loading cluster ID for follower catalog manager: success
I20260620 23:16:41.745563 1618 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 43d86874db344e2088e4017aeea2ccfb: acquiring CA information for follower catalog manager: success
I20260620 23:16:41.748246 1618 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 43d86874db344e2088e4017aeea2ccfb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:16:41.795800 1548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:41.780049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56534 (local address 127.10.191.253:42935)
0620 23:16:41.780275 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:41.780282 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:41.787630 (+ 7348us) server_negotiation.cc:407] Connection header received
0620 23:16:41.788162 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:41.788169 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:41.788405 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:41.788552 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:16:41.789307 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:41.790487 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:41.792641 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:41.793364 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:41.793465 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:41.794228 (+ 763us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:16:41.794968 (+ 740us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:16:41.795086 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:41.795337 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 23:16:41.795431 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
W20260620 23:16:41.838770 1544 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:16:41.829056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54070 (local address 127.10.191.252:36993)
0620 23:16:41.829294 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:41.829303 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:41.836647 (+ 7344us) server_negotiation.cc:407] Connection header received
0620 23:16:41.837172 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:41.837178 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:41.837316 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:41.837396 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:41.838500 (+ 1104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54070: BlockingRecv error: recv got EOF from 127.0.0.1:54070 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 23:16:41.888749 1548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:41.871247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56540 (local address 127.10.191.253:42935)
0620 23:16:41.871455 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:41.871462 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:41.881436 (+ 9974us) server_negotiation.cc:407] Connection header received
0620 23:16:41.882005 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:41.882012 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:41.882150 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:41.882230 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:16:41.882969 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:41.884123 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:41.886203 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:41.886995 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:41.887122 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:41.887793 (+ 671us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:16:41.887982 (+ 189us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:16:41.888053 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:41.888310 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 23:16:41.888402 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260620 23:16:41.933975 1544 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:16:41.924075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54082 (local address 127.10.191.252:36993)
0620 23:16:41.924335 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:41.924349 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:41.931871 (+ 7522us) server_negotiation.cc:407] Connection header received
0620 23:16:41.932395 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:41.932402 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:41.932526 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:41.932619 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:41.933732 (+ 1113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54082: BlockingRecv error: recv got EOF from 127.0.0.1:54082 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 23:16:41.988126 1548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:41.972263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56542 (local address 127.10.191.253:42935)
0620 23:16:41.972540 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:41.972550 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:41.980408 (+ 7858us) server_negotiation.cc:407] Connection header received
0620 23:16:41.980993 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:41.981002 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:41.981121 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:41.981205 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:16:41.981956 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:41.983307 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:41.985565 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:41.986324 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:41.986428 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:41.987133 (+ 705us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:16:41.987310 (+ 177us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:16:41.987384 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:41.987658 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 23:16:41.987755 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260620 23:16:42.035363 1544 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:16:42.025680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54088 (local address 127.10.191.252:36993)
0620 23:16:42.025895 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:42.025903 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:42.033353 (+ 7450us) server_negotiation.cc:407] Connection header received
0620 23:16:42.033896 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:42.033902 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:42.034001 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:42.034076 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:42.035109 (+ 1033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54088: BlockingRecv error: recv got EOF from 127.0.0.1:54088 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 23:16:42.084512 1548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:42.078080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56558 (local address 127.10.191.253:42935)
0620 23:16:42.078311 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:42.078318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:42.079590 (+ 1272us) server_negotiation.cc:407] Connection header received
0620 23:16:42.079839 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:42.079845 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:42.079945 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:42.080058 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:16:42.080533 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:42.081320 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:42.082732 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:42.083261 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:42.083404 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:42.083770 (+ 366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:16:42.083905 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:16:42.084003 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:42.084152 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 23:16:42.084230 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
W20260620 23:16:42.120855 1544 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:16:42.108779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54102 (local address 127.10.191.252:36993)
0620 23:16:42.109047 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:42.109054 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:42.119421 (+ 10367us) server_negotiation.cc:407] Connection header received
0620 23:16:42.119612 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:42.119616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:42.119706 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:42.119837 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:42.120627 (+ 790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54102: BlockingRecv error: recv got EOF from 127.0.0.1:54102 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
I20260620 23:16:42.138681 1548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:42.132861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56574 (local address 127.10.191.253:42935)
0620 23:16:42.133122 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:42.133132 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:42.133678 (+ 546us) server_negotiation.cc:407] Connection header received
0620 23:16:42.133907 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:42.133912 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:42.133984 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:42.134097 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:16:42.134639 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:42.135536 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:42.136854 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:42.137308 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:42.137447 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:42.137846 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:16:42.137966 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:16:42.138064 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:42.138223 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 23:16:42.138315 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 23:16:42.150555 1544 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:16:42.148272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54110 (local address 127.10.191.252:36993)
0620 23:16:42.148530 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:42.148539 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:42.149128 (+ 589us) server_negotiation.cc:407] Connection header received
0620 23:16:42.149342 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:42.149347 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:42.149432 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:42.149543 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:42.150296 (+ 753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54110: BlockingRecv error: recv got EOF from 127.0.0.1:54110 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 23:16:42.175184 1548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:42.169717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56590 (local address 127.10.191.253:42935)
0620 23:16:42.169964 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:42.169973 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:42.170510 (+ 537us) server_negotiation.cc:407] Connection header received
0620 23:16:42.170738 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:42.170743 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:42.170816 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:42.170969 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:16:42.171382 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:42.172157 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:42.173446 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:42.173911 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:42.174057 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:42.174452 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:16:42.174550 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:16:42.174643 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:42.174802 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 23:16:42.174917 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260620 23:16:42.183065 1544 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:16:42.181187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54126 (local address 127.10.191.252:36993)
0620 23:16:42.181383 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:42.181390 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:42.181599 (+ 209us) server_negotiation.cc:407] Connection header received
0620 23:16:42.181810 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:42.181815 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:42.181893 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:42.181996 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:42.182779 (+ 783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54126: BlockingRecv error: recv got EOF from 127.0.0.1:54126 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 23:16:42.243604 1548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:42.236675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56598 (local address 127.10.191.253:42935)
0620 23:16:42.236869 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:42.236880 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:42.237442 (+ 562us) server_negotiation.cc:407] Connection header received
0620 23:16:42.237653 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:42.237660 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:42.237755 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:42.237852 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:16:42.238390 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:42.239668 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:42.241734 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:42.242159 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:42.242258 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:42.242785 (+ 527us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:16:42.242999 (+ 214us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:16:42.243076 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:42.243259 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 23:16:42.243321 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 23:16:42.252269 1544 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:16:42.250332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54138 (local address 127.10.191.252:36993)
0620 23:16:42.250515 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:42.250522 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:42.250857 (+ 335us) server_negotiation.cc:407] Connection header received
0620 23:16:42.251071 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:42.251077 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:42.251185 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:42.251303 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:16:42.252025 (+ 722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54138: BlockingRecv error: recv got EOF from 127.0.0.1:54138 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 23:16:42.306725 1561 catalog_manager.cc:1259] Loaded cluster ID: 7df7cc41ccf44feca7728500695ced38
I20260620 23:16:42.306844 1561 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8046495c98464d89b87137568e1da7a7: loading cluster ID for follower catalog manager: success
I20260620 23:16:42.310844 1561 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8046495c98464d89b87137568e1da7a7: acquiring CA information for follower catalog manager: success
I20260620 23:16:42.313491 1561 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8046495c98464d89b87137568e1da7a7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:16:42.325029 1548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:42.319700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56614 (local address 127.10.191.253:42935)
0620 23:16:42.319942 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:42.319948 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:42.320184 (+ 236us) server_negotiation.cc:407] Connection header received
0620 23:16:42.320419 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:42.320425 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:42.320508 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:42.320607 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:16:42.321002 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:42.321774 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:42.323355 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:42.323775 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:42.323921 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:42.324298 (+ 377us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:16:42.324419 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:16:42.324515 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:42.324656 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 23:16:42.324735 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 23:16:42.336234 1544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:42.330551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54142 (local address 127.10.191.252:36993)
0620 23:16:42.330809 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:42.330816 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:42.331095 (+ 279us) server_negotiation.cc:407] Connection header received
0620 23:16:42.331316 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:42.331322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:42.331483 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:42.331605 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:16:42.331991 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:42.332707 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:42.334021 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:42.334447 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:42.334583 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:42.335147 (+ 564us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:16:42.335664 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:16:42.335833 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:42.335885 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 23:16:42.335965 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 23:16:42.349331 1548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:42.341674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56620 (local address 127.10.191.253:42935)
0620 23:16:42.341869 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:42.341877 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:42.342810 (+ 933us) server_negotiation.cc:407] Connection header received
0620 23:16:42.343126 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:42.343133 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:42.343232 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:42.343320 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:16:42.344035 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:42.345075 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:42.347316 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:42.347970 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:42.348095 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:42.348589 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:16:42.348736 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:16:42.348826 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:42.348986 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 23:16:42.349062 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 23:16:42.350003 1544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:42.342818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54152 (local address 127.10.191.252:36993)
0620 23:16:42.342998 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:42.343004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:42.343823 (+ 819us) server_negotiation.cc:407] Connection header received
0620 23:16:42.344105 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:42.344111 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:42.344188 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:42.344278 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:16:42.344898 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:42.345730 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:42.347893 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:42.348532 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:42.348662 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:42.349104 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:16:42.349243 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:16:42.349291 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:42.349685 (+ 394us) server_negotiation.cc:299] Negotiation successful
0620 23:16:42.349758 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 23:16:42.350148 1708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:16:42.341805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43636 (local address 127.10.191.254:39843)
0620 23:16:42.343138 (+ 1333us) server_negotiation.cc:206] Beginning negotiation
0620 23:16:42.343144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:16:42.343162 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:16:42.343272 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:16:42.343280 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:16:42.343486 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:16:42.343663 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:16:42.344317 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:42.345178 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:42.347545 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:16:42.348182 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:16:42.348312 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:16:42.349005 (+ 693us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:16:42.349443 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:16:42.349555 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 23:16:42.349756 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 23:16:42.349847 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1229,"thread_start_us":1135,"threads_started":1}
I20260620 23:16:42.355142 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 1326
I20260620 23:16:42.356084 1388 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:16:42.465128 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 1326
I20260620 23:16:42.485432 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 1400
I20260620 23:16:42.486555 1462 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:16:42.594172 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 1400
I20260620 23:16:42.615525 11007 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu pid 1474
I20260620 23:16:42.616603 1536 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:16:42.722352 11007 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpRJWBp/build/asan/bin/kudu with pid 1474
2026-06-20T23:16:42Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4282 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9129 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (359796 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST